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^This  study  investigated  the  definition  of,  and  causes  for,  success  in 

United  States  Air  Force  system  program  management.  The  major  activities 
were  to  (1)  construct  a  classification  of  success  criteria;  (2)  identify 
the  most  successful  Air  Force  system  programs  in  the  period  1965-1980;  and 
(3)  suggest  the  important  reasons  for  system  program  management  success. 

Data  was  gathered  through  (1)  a  survey  questionnaire  sent  to  current  and 
former  senior  Air  Force  and  Office  of  the  Secretary  Defense  oersonnel ; 
search  of  oficial  historical  and  congressional  records;  and  (3)  a  review  of 
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i^published  and  unpublished  literature.  The  study  concluded  that  system 
program  management  success  generally  is  defined  more  in  terms  of  how 
well  the  deployed  systeni  Performs  than  how  close  the  pj^oq ram  cost, 
schedule,  and  performance  requirements  were  met.  Lik^ise,"' the  leading 
causes  for  success  .tend  to  be  exoressed  by  factors  such  as  (1)  user 
understanding  of  the  system's  military  utility  and  (2)  how  well  the 
system  actually  worked.  The  renort  suggests  that  a  nroqram  manager,  to 
be  successful,  must  above  all  else  be  able  to  thoroughly  articulate  the 
system's  end  use;  with  this  understanding,  programmatic  structure  and 
decisions  can  be  properly  defined  ard  defended  through  every  level  of 
the  acquisition  organization.*,^^ _ 


SUMMARY 


A.  Research  and  experience  provide  convincing  evidence  that  no  magic  in¬ 
gredient  or  precise  mathematical  formula  has  yet  been  nor  will  ever  be 
found  to  guarantee  that  a  program  manager  will  succeed.  This  study  has 
concentrated  on  the  concept  "to  succeed,"  to  include  the  attributes  and 
imoortant  causes  of  success. 

S.  The  study  was  conducted  in  two  phases:  first  to  develop  a  classifica¬ 
tion  of  success  attributes,  then  to  use  the  attributes  to  identify  the  most 
successful  Air  Force  programs  in  the  1965-1960  time  period  and  prepare  a 
list  of  the  most  notable  causes  for  their  success.  The  desired  payoff 
of  the  study  is  that  a  knowledge  of  what  is  and  what  causes  success  will 
lirectly  assist  Air  Force  program  managers  in  developing  acquisition 
strategies  and  directing  their  programs. 

C.  PHASE  I 

1.  Phase  I  had  as  its  objective:  Defining  Success  Criteria.  The 
fundamental  question  was  "What  is  system  program  success?" 

2.  Two  working  hypotheses  formed  the  basis  for  investigation  of  the 
Phase  I  objective: 

a.  Hypothesis  I.  System  program  success  is  defined  in  terms  of 
'a  the  program's  record  in  meeti.ng  directed  cost/schedule/performance 
;C2P)  recuirements ,  and  (b)  the  program's  system's  ability  to  attain  and 
maintain  a  military  effectiveness. 


b.  Hypothesis  II.  The  closer  one  is  to  the  Program  Office  (in  an 
organizational  sense) ,  the  more  likely  it  is  that  success  will  be  defined 
in  terms  of  meeting  CSP  requirements. 

3.  All  research  was  directed  toward  the  following  overall  objectives: 

a.  Developing  a  classification  of  success  criteria. 

b.  Supporting/refuting  the  hypotheses.  - 

c.  Developing  a  list  of  major  programs  felt  to  have  been  success¬ 
ful. 

d.  Suggesting  possible  reasons  for  program  success  (as  inputs  to 
Phase  II). 

4.  Primary  research  was  conducted  through  the  use  of  a  mailed  survey 
questionnaire.  The  questionnaire  was  prepared,  tested  with  five  represen¬ 
tative  program  management  experts,  approved  by  Air  Force  Military  Person¬ 
nel  Center,  then  mailed  to  a  population  of  110  personnel  who  were  or  are 
currently  occupying  the  following  positions: 

o  HQ  USAF/RD/LE/XO  «  0  OSD/USDR&E 

0  HQ  AFSC/CC/CV  o  /ASD  (PA&E) 

o  ASD/ESD/SAMSO/CC  o  /ASD(C) 

o  Contractor  and  Air  Force  program  managers  from  ten  programs  from 

each  product  division  over  the  1965-1980  period. 

5.  Secondary  research  was  performed  through  literature  search  of  pri¬ 
vate  sector,  non-military  authorities  and  of  defense  (military)  related 
material  at  Washington,  D.C.  locations  such  as: 


American  University  Library 
Defense  Systems  Management  College  Library 
Federal  Acquisition  Institute  Library 
George  Washington  University  Library 


Industrial  College  of  the  Armed  Forces  Library 
Report  Abstracts  from  DLSIE,  DDC,  DTIC,  and  AU 


6.  The  study  resulted  in  these  findings: 

a.  Little  effort  is  made  in  management  literature  to  define  suc¬ 
cess  in  program  management.  Success  is  used  to  mean  fulfilling  "he  objec¬ 
tives  of  the  organization. 

b.  In  the  private,  non-military  sector,  program  success  usually 
is  equated  with  product  success.  The  basic  method  of  evaluating  a  program 
is  the  size  of  the  financial  return  from  the  resulting  product.  Hence, 
program  managers  are  very  knowleageable  of  and  sensitive  to  the  expected 
product  use. 


c.  Within  the  Air  Force  environment,  program  management  is 
described  by  AFSCP  800-3  as  a  process  of  completing  program  objectives. 
Some  authors  suggest  that  these  objectives  are  passed  down  to  the  program 
manager  as  his/her  "mission." 

d.  In  the  success  classification  developed,  the  criteria  under 
the  element  "System  Worked  Well  When  Needed"  were  listed  with  higher 
frequency  and  importance  than  "Accomplished  Program  Objectives." 

e.  Hypothesis  I,  "System  program  success  is  defined  in  terms  of 
(a)  the  program's  record  in  meeting  directed  cost/schedule/performance 
(CSP)  requirements,  and  (b)  the  program's  system's  ability  to  attain  and 
maintain  a  military  effectiveness,’'  was  substantiated.  The  respondents 
believed  that  a  system  program  is  judged  to  be  a  success  on  the  basis  of 
how  well  the  following  criteria  (in  order  of  importance)  have  been  met: 

( ’ )  The  system  was  operationally  capable. 

(2)  The  program  met  its  cost/schedule/performance  require¬ 


ments. 


(3)  The  program  maintained  stability  of  program  objectives. 

(4)  Deployment  of  the  system  was  timely. 

(5)  There  were  sufficient  numbers  of  the  system  to  be  mili¬ 
tarily  significant. 

f.  Hypothesis  II,  "The  closer  one  is  to  the  Program  Office  (in  an 
organizational  sense),  the  more  likely  it  is  that  success  will  be  defined 
in  terms  of  meeting  CSP  requirements,"  was  not  substantiated.  The  two  most 
important  factors,  irrespective  of  organizational  perspective,  appear  to 
be  (a)  Deploying  a  capable  system  and  (b)  Producing  a  system  for  which  the 
operational  requirement  has  been  clearly  established. 

g.  The  five  major  Air  Force  System  programs  felt  to  be  the  most 
successful  during  the  period  1965-1980  were: 

(1)  F-15 

(2)  C-141 

(3)  F-16 

(4)  Minuteman  II  and  III 

(5)  F-5E/F 

D.  PHASE  II 

1.  Phase  II  of  this  research  study  had  as  its  objective:  Determining 
Successful  Programs.  After  meeting  this  objective,  the  study  phase  con¬ 
sidered  the  fundamental  question  "What  are  the  reasons  that  certain  pro¬ 
grams  are  successful?" 

2.  The  two  working  hypotheses  which  formed  the  basis  for  investigat¬ 
ing  the  Phase  II  problem  and  fundamental  question  were: 

a.  Hypothesis  111  ;N0TE:  The  first  two  hypotheses  were  used  in 
Phase  I);  System  program  success  is  impeded  because  the  Prcg.^am  Manager 
is  disciplined  to  meet  Tost /Scheduls/Perfcrmance  \CSP)  requirements  and  to 
resist  changes  which  may  oe  necessary  to  ensure  long  term  military  effec- 


tiveness  of  the  system.  This  factor  causes  the  acquisition  management 
process  to  focus  on  the  program  objectives  instead  of  the  defense  weapon 
system  objectives. 

b.  Hypothesis  lY :  The  most  important  determinant  of  system  pro¬ 
gram  success  is  the  clarity  and  understanding  of  the  intended  milicary 
mission. 


2.  Research  was  conducted  by  the  use  of  an  investigative  worksheet 
using  historical  and  programmatic  records  at  HQ  USAF  and  HQ  AFSC.  Of  the 
list  of  iO  major  Air  Force  programs  during  the  1965-1960  period,  22  were 
evaluated,  and  lb  were  not  considered  because  of  program  age,  scope,  or 
security  classification. 

3.  The  study  results  led  to  the  following; 

a.  All  causes  of  success  were  classified  under  these  five  main 
.nput  parameters: 

(1)  Attention  to  directed  CSP  requirements. 

(2)  Ability  to  maintain  tight  control. 

(3)  Adequacy  of  means  to  accomplish  program  objectives. 

(y)  Support  for  the  system  program. 

(5)  Existence  of  clear  requirements. 


b.  Using  the  success  criteria  developed  in  Phase  I.  the  most 
successful  Air  Force  programs  of  those  evaluated  in  the  19b5-'9o0  perioc 
were  determined  to  be; 


Most  Successful 

Moderately  Successful 

C-14  1 

Gunship 

AGM-65  Maverick 

r-*5 

F-5E/F 

SRAM 

F-lb 

A-7 

E-3A 

A-'C 

KC-": 

V 


c.  The  Minuteman  II/III  program,  felt  by  the  survey  questionnaire 
respondents  to  be  one  of  the  most  successful  programs,  was  not  included  in 
the  top  two  success  classes  because  of  the  Minuteman  II  program  results. 
Rated  on  its  own,  the  Minuteman  III  program  would  have  been  included  in  one 
of  the  top  success  classes. 

d.  Important  causes  for  program  success  included: 

( 1 )  Availability  of  technology. 

(2)  Strict  adherence  to  system  performance/supportability . 

(3)  Funding  was  consistent. 

(4)  The  system  (program)  was  supported  by  HQ  USAF. 

(5)  Continuity  and  authority  of  the  program  manager. 

(6)  The  requirement  was  responsive  to  the  threat. 

(7)  The  contractor  demonstrated  excellence. 

(8)  Emphasis  on  change  management. 

(9)  Ability  of  program  office  to  maintain  tight  control. 

(10)  Attention  to  di.-ected  CSP  requirements. 

e.  Hypothesis  III,  "System  program  success  is  impeded  because  the 
Program  Manager  is  disciplined  to  meet  CSP  requirements  and  to  resist 
changes  which  may  be  necessary  to  ensure  lo.ig  term  military  effectiveness 
of  the  system,"  was  not  substantiated.  Collected  data  indicated  that  the 
desire  for  system  effectiveness  exceeded  that  for  adherence  to  CSP  re¬ 
quirements. 

f.  Hypothesis  IV,  "The  most  important  determinant  of  system  pro¬ 
gram  success  is  the  clarity  and  understanding  of  the  intended  mission," 
could  not  be  substantiated.  One  input  sub-parameter,  "Availability  of 
Technology,"  appeared  to  have  the  most  importance. 

g.  The  highest  priority  causes  of  system  program  success  are 
availability  of  technology  and  strict  adherence  to  system  performance.  A 
review  of  the  most  successful  programs  revealed  various  methods  to  enhance 


)  Prototype  to  demonstrate  technology. 

(2)  Extensive  operator  input  in  testing. 

'v3)  I'se  of  off-the-shelf  oomponents. 

(d)  Pollow-on  from  a  mature  system. 

Modification  of  a  mature  system. 

h.  System  program  success  tends  to  be  aeasurf>d  in  terms  of  how 

affirmative  and  strong  the  answers  are  to  the  following  questions; 

( '  '  Does  the  user  understand  what  to  do  with  the  system  to 
obtain  military  utility? 

\2)  loes  the  system  work  well? 

S.  Several  issues  were  raised  based  on  the  results  of  this  study; 

1.  Vhat  methods  are  used  within  the  program  office  and  by  the  acquisi¬ 
tion  management  review  process  to  estimate,  measure,  and  control  military 
:‘ility? 

2.  'fthat  will  be  the  effect  of  severely  restricting  entry  of  opera¬ 
tionally  experienced  personnel  into  the  acquisition  management  career 
field  because  of  personnel  shortages  ■' 

3.  Are  there  correctable  deficiencies  within  the  Air  Force  require¬ 
ments  process  which  impede  system  program  management  success? 
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"Philosophers'  shore.  An  imaginary  sucshance  rcoght  P^r  by  aloher- 
ists  in  hre  belief  bhob  ib  vcblc  orange  oase  seoals  inbo  golh  c.r.c  silver." 

The  cblective  of  bhis  ''esearo’r.  sbody  was  r.cb  t;  produce  a  "Philoco- 
p'r.ern'  rbore"  which  coui;  be  applied  in  i'usb  b.ne  rigr.b  acounbs  by  Air  Perce 
pregran  managers  to  ensure  success.  Basec  on  research,  and  experience,  it 
IS  safe  to  believe  that  nc  iiagic  ingredient  or  precise  lathenat ioal  formula 
■'.00  yet  beer,  or  will  ever  be  found  to  guarantee  that  the  progra.t  manager 
will  succeed.  This  st-dy,  instead,  has  ccr.cer.brated  on  the  concept  "to 

The  objective  of  every  rtar.ager  is  to  succeed.  This  assureptibn,  logi¬ 
cal  and  supportable  tc  the  p’ir.t  of  bei.ng  self-evident,  app*_es  in  every 
organisational  endeavor.  Air  Force  prograc  xanagers  face  the  p-esticn: 
how  do  I  know  if  I  iid  succeed?  The  answer — if  ':h.-:ir  prograx  was  success¬ 
ful— teccces  the  basis  and  reason  for  analysing  3ucces.s  in  systexs  pnegrax 
xanagexer.t . 


Air  Force  pregrax  xanagers  lack  their  private  sector  ccur.terpart' s 
all-pcwerful  "Botten  line."  Success  criteria  in  the  private  sector  are 
generally  expressed  in  terms  cf  financial,  return.  The  private  sector's 
p^oducb  (or  systex(  quickly  dexonstrates  its  aoility  tc  perferx  its  xis- 
sion ,  so  a  Judgment  of  the  program  (which  includes  the  product,  is  fairly 
straight-forward . 

In  Air  Force  system  program  management,  defining  success  requires 
consideration  cf  the  duality  of  purpose  faced  by  the  program  manager. 
First,  the  progi-am — that  is,  the  myriad  cf  activibies  to  complebp  a.a  acqui¬ 


sition  cycle — has  a  life  and  series  of  standards  cf  its  own.  .'■’bond.  t.:e 
■'esulting  system  has  success  c-iteria  totally  iivorced  from  “-.•..'e  cf  bne 
program.  It  Is  very  apparent  that  much  effort  is  expended  tc  increase  the 
manager's  ability  tc  sbcomplisr.  tne  program  r.ctbvbcy  tosKS.  .'r.-si.btr  bne 
amount  cf  trair.i''g  In  lefense  fystems  ’da.nagexent  Iol.e.ge  •.  'In  Force 


PAOS  BLANK -NOT  FILMS 


Institute  of  Technology  program  management  courses  on  financial  and  sched¬ 
ule  control.  Then,  consider  how  little  time  is  devoted  to  perfecting 
management  techniques  which  help  assure  that  the  resulting  system  will 
provide  the  necessary  military  utility.  How  many  program  managers,  fo^ 
example,  track  and  control  planned  system  capability  vs.  threat  with  the 
same  zeal  as  they  do  the  program  costs? 

There  is  a  need  to  identify  what  defines  success  in  system  program 
management  so  that  policies  and  procedures  which  enhance  success  can  be 
strengthened,  and  practices  which  inhibit  success  can  be  changed.  Program 
managers,  knowing  how  success  has  been  defined  (and  aided)  in  the  past,  can 
then  better  develop  and  implement  acquisition  strategies  which  will  help 
their  program  and  themselves  "to  succeed." 


4 


SECTION  I 


OBJECTIVES  AND  METHODOLOGY 


A.  PHASE  I 

' .  Objectives 

a.  Statement  of  the  Problem.  Phase  I  of  this  researcn  stiidy  had 
as  its  objective;  Defining  Success  Criteria.  The  funcamental  question 
was  '"rthat  is  system  program  success?" 

Care  must  be  taken  to  distinguish  between  cause  and  attribute . 
Phase  I  concerned  attributes ,  or  what  is  success.  Phase  II  of  this  study 
considered  the  causes  or  whys  of  success. 

The  word  "criteria"  in  the  statement  of  the  problem  is  used  in 
the  meaning  (Webster's  New  World  Dictionary,  ‘.966)  "A  test  by  which  a 
judgment  of  something  can  be  formed."  For  Phase  I,  the  challenge  was  tc 
defi.ne  and  prioritize  tnose  tests  to  which  system  program  activities  and 
results  can  be  subjected  so  that  a  worthwhile  value  judgment  may  be  made. 

b.  Hypotheses.  Two  working  hypotheses  formed  the  basis  for  in¬ 
vestigation  of  the  Phase  I  problem: 

(1)  Hypothesis  I.  System  program  success  is  defined  in  terms 
cf  (a)  the  program's  record  in  meeting  directed  cost/schedule/performance 
\CSP/  requirements,  and  (b)  the  program's  system's  ability  to  attain  and 
maintain  a  military  effectiveness. 

f 2)  Hypothesis  II.  The  closer  one  is  to  the  Program  Office 
(in  an  organizational  sense),  the  more  likely  it  is  that  success  will  be 
defined  in  terms  of  meeting  CSP  requirements. 

The  two  hypotheses  are  discussed  in  the  following  two  para¬ 
graphs: 


i 


1 


I 


t 
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(a)  Hypothesis  I  established  the  framework  for  the  ex¬ 
tended  classification  of  system  program  success  attributes.  This  classi¬ 
fication  is  discussed  in  Section  III. 

(b)  Hypothesis  II  viewed  the  two  broad  elements  of  sys¬ 
tem  program  success  from  the  perspective  of  ail  organizations  involved 
with  defense  system  acquisition  and  management.  Conceptually,  this  hypo¬ 
thesis  can  be  shown  as: 


Office 


System  Does 
Military  Job 


System  User 
(Operating  Command 
Or  OSD/JCS) 


This  conception  does  not  imply  that  the  Program  Man¬ 
ager  does  not  care  if  the  system  does  its  military  job,  or  that  the  user 
does  not  care  if  the  program  meets  its  CSP  requirements.  The  conception 
portrays  the  philosophy  of  "I'll  just  worry  about  my  own  job  and  let 
somebody  else  look  after  the  big  picture." 

c.  Research  Objectives.  All  research  was  directed  toward  the 
following  overall  objectives: 

(1)  Develop  a  classification  of  success  criteria. 

(2)  Support/refute  the  hypotheses. 

(3)  Develop  list  of  major  programs  felt  to  have  been  success¬ 
ful. 

(4)  Suggest  possible  reasons  for  program  success  (as  inputs 
to  Phase  II  of  the  study). 


IcM 


A 


The  population  list  is  at  Appendix  1. 

HQ  USAF/MPG,  rather  than  provide  the  addresses  of  most  active  and 
retired  generals  (to  ensure  privacy),  kindly  agreed  to  address  the  survey 
questionnaires.  Their  assistance  was  appreciated. 

Current  addresses  of  the  other  survey  participants  were  located 
and  used. 


(4)  Task  I-C:  Prepare  Survey  Format.  The  survey  question¬ 
naire  was  structured  to  provide  data  to  fulfill  the  research  objectives.  A 
dry  run  of  the  questionnaire  was  made  with  this  representative  sample  of 
Air  Force  program  management  experts  (all  0-6/GS-15  grade); 


Representing 
HQ  AFSC 
HQ  USAF 

Product  Division 

SPD 

OSD 


Position 
Assistant  DCS 
Exec  SAFAL  (RD4L) 
Former  SPD  at  ASD 
Former  SPD  of  ALCM 
Associate  Dean,  DSMC 


Comments  from  the  dry  run  were  incorporated  into  the  question¬ 
naire  format. 


Two  minor  problems  were  experienced  with  this  task.  First, 
there  existed  no  definitive  list  of  major  programs  from  1965-1980.  Through 
the  cooperation  of  HQ  AFSC/SDD  and  HQ  USAF/CVS,  the  list  was  compiled. 
Second,  obtaining  HQ  AFMPC/YPS  approval  for  the  survey  questionnaire  took 
almost  three  weeks  longer  than  the  planned  16  days. 

The  approved  Survey  Questionnaire  is  at  Appendix  2. 

(5)  Task  I-D:  Obtain  the  Survey  Data.  Questionnaires  were 
mailed  to  110  respondents  between  26  November  and  4  December  1980.  Respon¬ 
dents  returned  the  completed  questionnaires  in  stamped,  addressed  enve- 
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lopes.  A  follow-up  remirider  letter  was  sent  to  non-answering  respondents 
on  7  January  1981. 

Also,  secondary  research  was  conducted  in  Washington-area 
institutions.  Secondary  research/literature  search  activities  and  results 
are  described  in  Section  IlIA/3  of  this  report. 

(6)  Task  I-E:  Define  Success  Criteria.  The  task  required 
that  a  classification  of  success  criteria  be  developed,  and  a  list  of 
programs  believed  to  be  successful  be  prepared.  Section  III  discusses  two 
top  level  classifications  suggested  by  secondary  research,  and  the  classi¬ 
fication  required  by  the  study's  SOW. 

(7)  Task  I-F;  Prepare  Phase  Report.  The  data  and  results  of 
Phase  I,  plus  a  detailed  plan  for  Phase  II  were  provided  by  an  Interim 
Technical  Report  dated  and  delivered  31  December  1980. 

(8)  Task  I-FA;  Brief  Phase  I  Results.  The  Study  Team  Leader 
briefed  AFBRMC  personnel  at  Wright-Patterson  AF3,  OH,  15  January  1981,  on 
results  of  Phase  I. 

3.  Researoh  Methods 


a.  Primary  Research 

( 1 )  Survey  Questionnaire 

(2)  Personnel  Interviews  (Respondents  contacting  Study  Team 
Leader  if  desired). 

b .  Secondary  Research 

(1)  Literature  search,  Non-defense  private  sector. 

(2)  Literature  search.  Defense  sector,  including  unpublished 


research  reports  and  case  studies. 


B.  PHASE  II 


I ,  Obj  ect Ives 

a.  Statement  of  the  Problem 

Phase  II  of  this  research  study  had  as  its  objective:  Deter¬ 
mining  Successful  Programs.  After  meeting  this  objective,  the  study  phase 
considered  the  fundamental  question  "What  are  the  reasons  that  certain 
programs  are  successful?" 

Results  of  this  study’s  Phase  I  have  provided  a  classification 
of  system  program  management  success.  This  listing  became  the  criteria  to 
apply  to  major  Air  Force  system  programs  so  that  a  rank  order  of  successful 
programs  could  be  made.  Having  identified  the  most  successful  programs, 
the  study  effort  attempted  to  establish  causal  relationships  between  man¬ 
agement  inputs  and  demonstrated  program  success. 

b.  Hypotheses 

The  two  working  hypotheses  which  formed  the  basis  for  investi¬ 
gating  the  Phase  II  problem  and  fundamental  question  were: 

(1)  Hypothesis  III  (NOTE:  The  first  two  hypotheses  were  used 
in  Phase  I):  System  program  success  is  impeded  because  the  Program  Mana¬ 
ger  is  disciplined  to  meet  Cost/Schedule/Performance  (CSP)  requirements 
and  to  resist  changes  which  may  be  necessary  to  ensure  long  term  military 
effectiveness  of  the  system.  This  factor  causes  the  acquisition  manage¬ 
ment  process  to  focus  on  the  program  objectives  instead  of  the  defense 
weapon  system  objectives. 

(2)  Hypothesis  IV:  The  most  important  determinant  of  system 
program  success  is  the  clarity  and  understanding  of  the  intended  military 
mission. 
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Diacusslon  of  Hypotheses 


Hypotheses  III  and  IV  are  related  through  the  idea  that  with¬ 
out  a  concise,  clear-cut  knowledge  of  desired  system  use,  the  System  Pro¬ 
gram  Manager  cannot  understand  nor  endorse  changes  to  the  program  base¬ 
line,  so  adopts  a  position  of  defending  the  status  quo.  This  position 
finds  meaning  in  the  expression  "It's  their  (the  operator's)  system,  but 
it's  ay  program."  Hence,  any  proposed  change  becomes  viewed  as  a  threat  to 
the  program  (and  program  manager). 

The  study  phase  used  Hypotheses  III  and  IV  in  the  process  of 
determining  what  programs  were  successful  and  why.  If  collected  data  could 
substantiate  the  two  hypotheses,  the  conclusions  should  show: 

( 1 )  System  Program  Managers  believe  their  programs  CS?  re¬ 
quirements  are  their  personal  standards  of  success. 

(2)  System  Program  Managers  can  and  will  enthusiastically  ef¬ 
fect  necessary  changes  to  programs  only  when  the  resulting  improvement  of 
the  system  to  do  its  mission  can  easily  be  perceived. 

(3)  The  simpler  the  system  military  mission  can  be  explained, 
the  more  support  the  whole  bureaucratic  organization  will  give  the  program 
and  program  manager. 

2.  Methodology  of  Study  Phase 

a.  Task  Activities 


(1)  Task  II-A:  Prepare  Program  Population.  Using  data 
available  at  HQ  USAF  and  HQ  aFSC  (the  significant  data  sources  for  Phase 
II),  a  listing  was  prepared  of  all  USAF  major  programs  which  have  experi¬ 
enced  a  DSARC  III  (or  equivalent)  milestone  f^om  ‘  July  1955  through  30 
April  1950,  and  which  resulted  in  definable,  deployable  weapon  systems. 
See  Section  IV-3  for  3  discussion  of  tnis  task  list. 


(2)  Task  II-B:  Prepare  List  of  Input  Parameters.  A  list  of 
input  parameters  was  determined  by  research  of  each  population  program  and 
Phase  I  results.  This  list  was  reviewed  by  and  coordinated  with  AFBRMC, 
effective  23  January  1981. 

(3)  Task  II-C:  Obtaining  Input  Parameter  Data.  Each  popu¬ 
lation  program  was  analyzed  to  obtain  the  input  parameter  data  as  specified 
in  Task  II-B. 


(4)  Task  II-D:  Ranking  Programs  Into  Success  Classes.  Each 
program  was  analyzed  using  the  Success  Criteria  determined  in  Phase  I.  The 
analysis  resulted  in  categorizing  each  program  into  Success  Classes. 

(5)  Task  II-E:  Compare  Successful  Program  Rankings.  The 
programs  in  the  top  two  Success  Classes  of  Task  II-D  were  compared  with  the 
"list  of  programs  felt  to  be  successful"  from  Task  I-E.  Significant 
anomalies  are  discussed  in  Section  IV-C. 

(6)  Task  II-F:  Analyze  Input  Parameters  for  Successful  Pro¬ 
grams.  From  the  list  of  programs  in  the  upper  Success  Classes  (Task  II-D) 
as  generally  validated  by  the  Task  I-E  list,  each  group  of  input  parameters 
was  analyzed  to  determine  those  input  types  which  appear  to  be  the  most 
correlatable  with  program  success. 

(7)  Task  II-G:  List  Input  Factors  Associated  with  Successful 
Programs.  The  input  parameters/factors  (both  controllable  and  uncontrol¬ 
lable)  which  appear  to  correlate  with  program  success  were  identified  and 
are  listed  in  Section  IV-D. 

(8)  Task  II-H:  Prepare  Phase  Report.  The  data  and  results 
for  Phases  I  and  II,  with  hypotheses  results  described,  has  been  prepared 
as  a  Final  Report.  The  ADTECK  Study  Team  Leader  shall  brief  AFBRMC  person¬ 
nel  at  Wright-Patterson  AFB,  OH  on  results  of  this  report.  The  briefing 
shall  be  modified  as  directed,  then  formally  presented  to  representatives 
of  AFRDC  and  AFSC. 


t; 
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3.  Research  Method 


The  basic  method  used  for  research  during  this  study  phase  was 
search  of  official  records.  Histcrical  and  programmatic  documentation  at 
'<Q  j’SAF  and  HQ  AF.SC  included  the  following  types  of  official  records: 

a.  Selected  Acquisition  Reports  (SAR). 

b.  Official  Histories. 

c.  System  Program  Monographs. 

d.  Program  Assessment  Review  (PARI  brief ings. 

e.  Published  and  unpublished  literature. 

(NOTE;  This  report  contains  no  quantitative  program  data  cf  any 
type. ) 


SECTION  III  ATTRIBUTES  OF  SUCCESS 


Overview.  This  section  responds  to  Section  C.4.1  of  the  SOW,  and  expands 
the  findings  of  Interim  Technical  Report  F336 '^5-80-0-5184-1  to  include  all 
survey  questionnaires  received  through  13  February  1981.  The  discussion 
order  for  this  section  is  success  in  the  non-military  sector,  success  in 
the  military  (defense)  sector,  Hypotheses  I  and  II  (to  include  a  classifi¬ 
cation  of  success  criteria),  and  some  overall  observations  concerning  suc¬ 
cess  criteria. 

A.  Success  in  the  Private,  Non-Military  Sector. 

1.  Research  purpose:  Through  secondary  research,  determine  the  views 
of  private  sector,  non-military  authors  on  successful  or  effective  program 
management.  Sub-objectives  included: 

a.  Estimate  the  importance  of  program  n:anagement  in  the  total 
management  environment. 

b.  Seek  case  study  examples  as  contrasts  between  unsuccessful  and 
successful  programs. 

c.  Through  the  literature,  compare  the  emphasis  placed  on  the 
project  itself  vs.  the  resulting  product. 

This  seconda"y  research  was  conducted  through  a  review  of  books 
and  periodicals  at  the  following  locations; 

American  University  Library 
Andrews  AFB  Base  Library 

Defense  Systems  Management  College  Library 
Federal  Acquisition  Institute  Library 
George  'Washington  University  Library 

Headquarters  Air  Force  Systems  Command  Technical  Irfcrmaticn 
Center 


Industrial  College  of  the  Armed  Forces  Library 

Report  Abstracts  from  DLSIE,  DDC,  DTIC,  and  Air  University 

All  information  facilities  were  searched  using  these  key  words; 

Management,  Program 
Managemen t,  Project 
Program  Management 
Project  Management 
Success 

Systems  Management 
Weapon  System  Management 

2.  Research  findings:  Views  of  success  in  the  private,  non-military 
sector. 


a.  Importance  of  Program  Management.  Program  management,  in  a 
non-military  environment,  is  not  the  ordinary  mode  of  management  (func¬ 
tional  management  is).  Also,  the  term  "program”  is  sometimes  used  inter¬ 
changeably  with  "project,”  and  at  other  times  used  to  mean  a  long-term 
undertaking  made  up  of  more  than  one  project.  The  output  of  a  program  is, 
for  the  most  part,  a  product.  Hence,  the  three  terms,  program-project- 
product,  may  be  referring  to  the  same  process.  For  definitional  purposes, 
there  is  some  justification  to  show  that  program  management  encompasses 
the  project  activities  which  result  in  a  product.  The  remainder  of  this 
report  will  use  the  definition  for  the  private,  non-military  sector  that  a 
program  consists  of  projects  which  yield  a  product. 

The  non-military  program  manager,  while  concerned  in  the  short 
run  about  project  progress,  is  expected  to  stay  with  the  program  until  the 
product  either  performs  or  fails  to  perform  its  mission.  Hence,  the  non- 
military  program  manager  devotes  considerable  attention  to  understanding 
the  product's  expected  performance.  Because  of  this  knowledge  of  the 
product's  goal,  the  non-military  program  manager  is  expected  to  define  the 
standards  of  performance  which  will,  in  the  end,  determine  if  the  proauct 
(hence  the  program)  was  a  success. 
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Program  management  remains  an  exception  within  tne  ncn-miii- 
tary  environment  because  of  three  primary  reasons:  First,  it  is  r.ct 
normaily  neecec.  Companies  grow  up  using  their  predominant  prccuot  as  the 
main  focus  of  organicaoicnai  development .  Program  management,  with  its  ac 
hoc  connotation,  is  therefore  not  needed.  Second,  it  is  inefficient. 
Program  ma.nagement  decreases  the  functional  manager's  ability  to  ittimaiiy 
use  the  existing  personnel  resources.  Third,  few  personnel  are  trained  cr 
experienced  ti  beicme  et’fective  program  managers.  The  genera-ist  is  a 
rarity  as  rewards  witnin  a  ccmpa.ny  enccurage  functional  speciaiitation . 
.Additionally,  program  management  training,  as  such,  receives  little  sup¬ 
port  . 


b.  Case  Study  Examples. 

\1}  Ford  Motor  Company;  Fcsel  vs.  Mustang.  The  sticy  team 
predicted  that  this  classic  management  comparison  would  be  an  ideal  exam- 
ole  of  project  vs.  product  success.  The  team  fe.t  that  data  cou..  :  le  i  our.d 
to  discuss  the  two  programs  in  term.s  of  project  estimated  vs.  actual 
cost/schedule/performance  (CS?)  as  indications  of  success.  .Accessable 
literature  did  not  substantiate  this  avenue  of  investigation,  .however. 
All  comparison  involved  the  products — why  Fdsel  as  a  comucercial  venture 
failed  and  .Mustang  succeeded.  In  this  case  example,  the  products’  •  issions 
were  identical:  .sell  many  units)  to  provide  to  the  company  a  sucst'.ntiai 
financial  return.  The  project  CS?  requirements  die  not  ever,  rate  a  foot¬ 
note  in  history. 

\2'i  Motion  pictures:  Twentieth  Century-Fox  "Sta’"  Wars"  vs 
Thitsd  Artists'  "Heaven's  Gate."  The  r.cst  ‘'inancially  successful  metior. 
cicture  in  history,  "Star  'wars,"  was  over-budget  when  compared  with  origi¬ 
nal  estir.ates;  by  definition  an  unsuccesst  ui  project.  "Heaven's  l.ate," 
whose  production  ocst  escalated  from  its  origir.ni  budget  of  $11. o  m.iilicn 
to  over  million,  was  liAewise  a.n  unsuccess:'ui  project  and  h.a/u.g  beer, 
withdrawn  from  distribution)  was  neither  .t.n  artistic  nor  iirar.b.ai  suc¬ 
cess.  "Star  Wars''  was  a  spectacularly  sucoesodul  orogram;  '’Henv*^"  o  Tate" 
a  spectactular  failure. 


Motion  piotufg  tlthifeiiS  iASi^  «r  pfogt^ 
mantgofflonti  single  pfdiuett  finite  eunetien  te  ^pedtieev  aftd  'ani^a^  ^ipii- 
retion  requirtBonte  (ielSi  leealiengt  end  peneehhei)-.  Sd'e^e^iMi  film 
produotion  ss  pfogrea  eueeeee  le  penei^  pipeetea  eh  the  hehih  oi'  iteetihg 
cost  or  soheOuie  estilaiest  ¥h§  iiaaaPeS  of  ^uSefeas  aPe  fihaheial  rotorn, 
artiatio  aoolaia,  op  pooepiien  of  ewaPae  hy  the  fiii  iteeif-.  ideal, 
aoat  suooessful  sotion  piOluPO  trOtiiO)  in  fchghp^i  daikiaite  ell  three  «»e- 
3ur«s  (“CJone  With  tfie  Wind”  Cdaes  to  mihd),  yet  still  meet  CSP  goals.  If 
CSP  goals  are  missed  and  product  success  is  gained,  the  program  is  a 
success.  Henoe,  program  success  equals  product  succesisv 


The  failure  of  "Heavoh'S  Qate**  was  not  simply  that  it  came  ir. 
3i  times  over  budget,  but  that  it  did  not  succeed  in  any  of  the  three 
measures  of  suooesa.  Ah  expected  outcbSe  of  the  "Heaven's  Gate"  experi¬ 
ence,  reports  one  recent  article  (2426),  is  that  in  the  future  the  direc¬ 
tors  will  be  held  accountable  for  their  budgets;  if  sb,  the  interesting 
question  which  then  cofcOS  td  Sind  is:  Wbuld  '•'HBai^h's  ^te"  have  been  a 
success  if  it  had  coffle  id  oh  sCheddiB  for  ill; 6  dillioh?  If  the  same 
product  had  resulted  (ho  fihandiai  rdfciiPh,  artistic  dheiaii,  or  a'waras), 
the  answer  certainly  sdems  to  be  Me;  The  producer  (closest  eqaivai'ent  to 
program  manager)  who  tuPhs  out  a  •'turkey**  fails,  no  matter  how  close  to  the 
CSP  goals  he/she  comes. 

0.  Project  vC  Product  Emphasis;  Although  thfe  case  examples  sug¬ 
gest  that  product  success  is  paramount j  there  was  some  ihforBAtioh  in  the 
reviewed  literature  which  provided  techhiques  for  project  management. 
Success  Was,  in  ofte  instance,  implied  is  the  development  of  i  system  (or 
product)  of  the  highest  tedhhicai  quality,  developed  it  the  idWest  cost 
possible,  in  the  shortest  possible  time;  Assuredly,  thiS  would  be  ah  ideal 
situation. 

Other  sources  suggested  that  the  manager  was  responsible  for  de¬ 
fining  the  o’bjeetiVes,  the  standards  of  Success,  then  the  tasks  by  which 
the  objectives  were  to  be  achieved;  Mb  insight  was  given  to  what  Succ’ess 
meant;  instead/  causes  for  Success  were  listed;  TheSe  ciuSiS  'w%r%  'w%ii 
disoussed:  prograS  definition ,  careful  plahhihg,  ahd  idi^uiti  cohtrhl'. 
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d.  Overall  Observations.  Literature  or.  "How  to  Succeed"  revealed 
little  insight  on  how  to  define  Program  Management  success.  This  seemed  to 
be  self-obvious;  it  was  what  the  organization  decided  it  should  be.  Pro¬ 
ject  management  techniques  to  meet  CSP  goals  were  described,  and  effective 
(or  successful)  management  was  discussed  in  terms  of  "how  to"  rather  than 
"what  is."  The  consensus  position  seemed  to  be  that  project  (i.e.,  activi¬ 
ty)  success,  while  important,  mattered  much  less  than  how  well  the  product 
fared. 

A  top  level  success  classification  for  non-military  programs  follows: 


Success,  Non-Military  Programs 


F inancial 
Return 


Artistic 

Acclaim 


Technology 

Advance 


Service 


B.  Success  in  the  Defense  (Military)  Sector 

1.  Secondary  Research.  An  important  source  of  information  concerning 
success  in  systems  program  management  was  from  secondary  sources.  The 
purpose  was  to  search  the  defense  (military)  related  literature  to  deter¬ 
mine  how  success  is  defined,  discussed,  or  suggested.  This  search  comple¬ 
mented  the  primary  research  because  both  secondary  and  primary  research 
information  c«ime  from  the  same  population  group:  defense  program  manage¬ 
ment  experts.  The  research  investigated: 

Published  literature.  Information  in  this  category  tended  to 
be  polished  and  sanctioned,  reflecting  an  officially  approved  aura. 

Unpublished  literature.  This  information,  though  not  as  re¬ 
fined,  reflected  more  of  how  the  process  really  works  as  opposed  to  how  it 
is  supposed  to  work. 

a.  Research  locations. 

The  literature  search  was  conducted  at  the  sources  listed  in 
III.A.l  (above)  as  well  as: 

Air  Force  Association  Headquarters  (Periodicals) 

Unpublished  student  reports  at: 

Defense  Systems  Management  College 
Industrial  College  of  the  Armed  Forces 
National  War  College 

b.  Research  findings: 

(1)  Program  Management  is  Important.  There  is  no  doubt  that 
program  management  is  important  in  the  defense  environment.  Program  man¬ 
agement  has  synonyms:  project  or  systems  management.  The  previous  "pro¬ 
gram  equals  project  plus  product"  definition  used  in  the  non-military  area 
finds  no  counterpart  here.  Most  writ  ngs  use  the  terms  program,  project, 
and  systems  as  equivalents  when  describing  program  management.  This  may  be 
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important  in  defining  success,  because  though  used  intcrohangeaoly ,  the 
terms  program  and  system  are  not  the  same  if  the  system  means  the  resulting 
defense  weapon  system.  Air  Force  program  management  is  definec  (AF3CP  cGC- 
j;  as  "The  process  whereby  a  single  manager  is  responsible  for  planning, 
organizing,  coordinating,  directing,  and  controlling  the  combined  efforts 
of  Air  Force  contractors  and  participating  organizations  in  accomplishing 
program/ pro ject  objectives."  Successful  program  management  accomplishes 
orogram/ pro Ject  objectives  to  deplo2/  a  system.  These  program  objectives 
are  different  from  the  program's  product  or  system  objectives. 

(2)  Definitions  of  Success  are  Rare.  The  literature  contains 
many  works  on  how  success  in  program  management  is  to  be  achieved  through 
organization,  technique,  or  managerial  qualities.  Usually  the  definition 
of  success  is  absent  with  no  reason  being  given  for  the  omission.  Two 
authors  recognized  a  need  to  consider  the  meaning  of  success,  but  settled 
with: 

"...If  a  successful  program  manager  is  one  who  managed  a  suc¬ 
cessful  program,  then  a  precise  quantification  of  a  successfu.  program 
becomes  important.  Unfortunately,  i.n  most  instances  there  is  no  standard 
for  comparision;  for  no  two  programs  are  sufficiently  alike  in  span,  in 
time,  or  in  content  to  permit  direct  comparison.  Simply  stated,  ?  success¬ 
ful  program  is  one  that  did  not  fail  (2.20)." 

"I  will  not  attempt  to  define  what  a  successful  program  mana¬ 
ger  is. . ."  (2.7) . 

A  recent  report  published  by  Air  Force  Systems  Command  (2.1) 
acknowledged  that  if  the  objective  is  to  have  a  successful  program,  then  a 
logical  question  is  to  ask  what  sta.ndards  or  measurements  determine  whe¬ 
ther  a  program  is  successful.  Bes.des  meeting  CS?  goals,  ten  onaracter- 
. sties  were  suggested; 

Recognized  need 

Management  commitment 

One  individual  responsible  with  authority 


Sound  technical  foundation 
Risk  management 

Comprehensive  and  dynamic  planning 
Good  contractors  with  recent  experience 
Realistic  requirements  and  schedules 
A  firm  baseline  with  traceability 
Capability  available  when  needed 

Only  the  last  characteristic— capability  available  when 
needed — is  an  effect  or  output  of  a  program.  The  other  nine  characteris¬ 
tics  represent  causes  or  inputs.  One  very  precise  definition  of  success 
was  found  in  a  paper  (2.10)  describing  the  Mavericx  AGM-65  program; 

"Among  indications  of  the  program's  success  were  the  attain¬ 
ment  by  Hughes  of  all  maximum  positive  performance  incentives,  the  program 
receipt  of  the  prestigious  Daedalian  Award  and  the  attainment  of  a  total  of 
four  stars  by  three  former  Air  Force  Maverick  SPO  Directors." 

A  description  of  the  F-15  program  (2.3)  contained  an  inclusive 
goal  for  success:  To  build  the  best,  on  time,  and  within  budget.  This 
perspective  places  capability  of  the  resulting  product  ahead  of  meeting 
the  project  goals  in  importance.  Taking  this  perspective  one  step  further 
was  the  former  Director  Defense  Research  and  Engineering,  Malcolm  R. 
Currie.  In  prepared  remarks  to  students  at  the  Defense  Systems  Management 
College  (2.8),  Dr.  Currie  acknowledged  the  Importance  of  adequate  program 
definition  and  of  using  cost  as  a  valid  engineering  design  requirement,  but 
concluded  with  his  belief  that  "There’s  only  one  final  Judgment  on  the 
quality  of  a  program.  That  is  how  well  the  defense  system  works  in  the 
field  when  it  is  needed." 

(3)  Success  is  Usually  Equated  with  Meeting  Program  Objec¬ 
tives.  Program  management,  by  definition,  is  a  process  to  accomplish 
program/project  objectives.  Little  wonder,  then,  that  most  authors  equate 
successful  program  management  with  attaining  objectives.  The  question 
becomes:  What  are  the  program  objectives? 


It  13  important  to  renemoer,  when  considering  tne  derivation 
r''  nrograa  objectives,  that  progras  rnanageaer.t  for  defense  systens  occors 
i.h  the  military.  He.nce,  all  officers  are  bound  by  oath  to  do  i.^ei"  uimcst 
■■  j  ach'eve  promct  and  sucoes"‘'ul  cottpleiicn  of  the  mission,  an  i  '  ■'e  "i.ss.on 
.and  assigned  ''y  military  suooriors.  ra.oic  p-cgram  ‘■  ■'ctlves 
-ecome  those  wh:oh  are  established  in  the  charter.  In  "16;.  '  ■>  'n>-r. 


-;reEGr.  -  ":=an£ 


same  thing  that  it  does  in  ccocat — prompt  resp  ,nse 


rders...."  In  ’9b9,  the  ocnmander  of  .^F3C  said  .2.'^^.:  "...tne  System 

'"Ograin  Jirector's  ^ob  is  management  cf  a  program.  li  is  not  tc  ouy  the 
'est  equipment  that  can  be  definea  as  the  program  deve.cpaent  proceeds;  it 
p  to  buy  what  has  beer,  approved,  within  the  costs  mat  nave 
.'dtec,"  T.'it  notion  cf  defining  pregr.ar.  ftjert.  .'o.;  •  the  ;'-.g:'a.- 
rontinues  to  the  present,  as  described  in  jeteber  '979  ,1.5/':  Cnee  clear 
■  nd  well-defined  cost,  schedule,  and  performance  baselines  are  estao- 
ished,  program  directors  cannot  change  them  witncut  a  forma,  a. ‘her; ting 
o'^un6''  • 


In  addition  to  the  fact  that  program  oojectives  are  directed 
■‘.own  the  chain  of  command,  and  thus  become  tne  goa.s  to  wnicn.  pregra.m 
activities  are  directed,  CS?  goals  tend  to  be  the  objectives .  'hese  CSh 
ibjectives  retain  ■widespread  use  because  they  car;  b-^  measured.  '■'.lita’"/ 
tanagers  lack  the  profit  motive  arc  other  easi.y  duantif i3Cl“  measures  cf 
’’Ucoess  of  the  resulting  system  '.or  product;  [2.21).  "Fesults  in  cost, 
■c.redule,  and  performance  have  been  bhe  .standard  rreas-ur-eo  of  succt  rs  in  all 
our  methods  of  managing  defense  systems  acquisiticn.  Such  resu.ts  are 

-nceed  the  tell  tale  yardstic<.s -  I’-ey  d^.-ect  attention  to  a  quar.tita- 

'  ive  scale  of  success  in  terms  cf  money,  time,  .^nd  er.gineering  units" 
2.2’'.  It  is  easy  tc  conclude  that  program  objectives  mean  to  most 
m. litany  program  managers  those  iSt  goals  ■which  car.  and  will  be  m.easurec  bv 
b.aeir  military  suoeric''S  within  the  'acquisition  ■'ommunity. 

(u)  Change  is  tc  be  avc.ded.  ’'cc.ay'.^  .-a? 50  progra.’.  manage  -s 
operate  wibn  a  polic/  '.hat  oase'irt’o,  -nc'^  -r  tac -ir''e  d ,  can  be  changed 
only  by  formal  d.rec..icn,  'iveaccn  sy.-’';ems  'cni'-acts  and  tne  pricing  system 
■■-nd  tc  be  based  -n  “■e  •-u-,.  .  ■'  . 


Design  change  requirements  may  be  equated  (2.1b)  with  problems  which  are 
the  "seeds  of  overruns  and  degraded  capability."  An  early  study  of  the  F- 
15  program  success  (2.22)  gave  the  philosophy  "Assuming  that  the  job  of 
defining  goals  and  implementing  them  into  lean  specifications  is  complete, 
there  should  be  no  need  for  changes....  Once  a  program  baseline  is  estab¬ 
lished,  changes,  while  often  tempting,  should  be  resisted."  The  avoidance 
of  change  leads  to  more  detailed  definitions  of  program  plans,  which  in 
turn  leads  toward  airtight  contracts  and  responsibility  agreements  in 
which  the  contractors  and  Government  seem  more  concerned  with  protecting 
themselves  from  each  other  than  in  obtaining  optimum  systems  within  the 
constraints  of  time  and  the  budget  (2.9).  If  change  is  to  be  avoided, 
then,  stability  is  desirable;  it  becomes  a  key  (or  perhaps  a  definition)  of 
program  success. 

(5)  Top  Level  Classification  for  Military  Programs:  This 
classification  has  been  suggested  through  the  secondary  research: 


Success 


Accomplishing  I 

Program  Objectives  Kewards 


System  Works 
Well  When  Needed 
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2.  Primary  Research.  The  primary  research  instrument  was  a  survey 
questionnaire  .Appendix  2}  mailed  to  the  survey  population  listec  in  Ap¬ 


pendix  ’ 


a.  Information  concerning  the  questionnaire. 


)  Population.  The  survey  population  was  select  ad  Cased  :n 


the  following  factors; 


,a)  Respondents  had  to  be  in  decision-making  positions 
with  a  broad  view  of  the  defense  acquisition  process. 


(b)  The  number  of  desired  responses  was  tC-rC.  The 
answer  population  was  1K',  and  a  bTi  response  rate  from  personnel  of  the 
proven  responsibility  of  this  group  appeared  reasonable. 


(c)  The  orientation  of  the  population  was  predominately 
that  of  the  Program  Manager  because  the  Study  Team  believed  -hat  the 
Program  Manager  would  have  the  clearest  and  most  deeply  held  cp.nion  tf  the 
definition  of  system  program  success. 


(NOTE;  Population  respondents  were  furnished  the  Stucy  Tear 
leader's  name  and  telephone  number  if  any  additional  dialogue  was  de¬ 
sired  . ) 


(2'  Quest! or.r.a i re  T t rue t ure .  The  questionnaire  'Appendix  2; 
was  composed  of  five  parts.  Hespcncents  were  requester  to  complete  each 
part  in  turn,  not  retu.’'r.ing  t^  completec  parts.  This  instruction  was  tc 
avoid  biasing  responses  tc  Parts  l-III,  because  Part  TV  contained  ten 
factors  which  could  h.ave  been  interpreted  as  the  "answers''  to  what  is 


system  program  success, 
search  intent  of  each  ques 


V4J0  r'G” 


Purposes: 


2  Require  respondents  to  consider  the  overall  ques¬ 
tion  of  system  program  management  success. 

^  Define  system  program  management  success. 

2  Substantiate/refute  Hypotheses  I  and  II. 

(b)  Part  II.  Requirement  for  specific  choice  of  most 
successful  programs,  with  reasons. 

Purposes ; 

2  List  attributes  of  successful  system  programs. 

2  Identify  programs  felt  to  be  successful. 

2  Substantiate/refute  Hypothesis  I. 

vc)  Part  III.  Requirement  for  specific  choice  of  least 
successful  programs,  with  reasons. 

Purposes ; 

2  List  attributes  of  unsuccessful  system  programs 
(a  method  of  negatively  defining  what  is  success). 

2  Identify  programs  felt  to  not  be  successful. 

(d)  Part  IV.  Forced  ranking  of  factors  whicr.  define 
system  program  success. 

Purpose:  Substantiate,- refute  Hypothesis  II. 


merit  at  the  Defense  Systems  Management  College,  in  a  recent  paper  "The 
Interface  Between  the  DOD  Manager,  the  OSD  Policy  Maker  and  the  Acquisition 
Researcher  -  A  Study  of  Management  by  Compulsion,"  stated  his  belief  that 
"Most  acquisition  research  fails  to  take  the  industrial  view  into  ac¬ 
count...."  The  response  results  from  this  study  indicate  that  industry 
involvement  may  not  improve.  Reasons  for  the  lower  involvement  could 
include: 


2  One  respondent;  "It's  corporate  policy  to  not 
respond  to  these  type  surveys.  They  have  to  be  answered  through  the 
corporation  headquarters." 

2  Reluctance  to  provide  any  information  which  may 
appear  to  be  derogatory  toward  important  customer  (Air  Force). 

^  Indifference  to  any  activity  which  will  not  lead 

to  a  financial  return. 

(c)  Reaction  to  questionnaire.  Four  specific  comments 
concerning  the  effectiveness  of  the  survey  questionnaire  were  received: 

2  Telephone,  23  Jan  8l,  previous  Vice  Commander, 
AFSC;  "The  questionnaire  could  not  allow  me  to  express  the  frustrations 
which  were  encountered  in  trying  to  do  the  job,  but  not  being  allowed  to  by 
OSD  and  Air  Staff  people." 

2  Telephone,  5  Feb  3l,  previous  DCS/RD,  Air 
Staff:  "The  questionnaire  struck  me  as  another  search  for  the  magic  form¬ 
ula  to  program  management.  And  there's  not  one." 

2  better  accompanying  returned  questionnaire,  22 
Dec  SC,  previous  SPD  on  two  major  ASD  programs:  "The  questionnaire  is 
provocative  but  I  am  of  the  opinion  that  there  should  oe  face-to-face 
dialog  on  such  research.  There  are  simply  too  many  facets  for  examinatio 


^  Letter  accompanying  returned  qcest.  ,n:.c -'■c,  L 
r'eb  0’,,  from  previous  DDR&E;  "In  general,  I  feel  that  your  cucstior.ra-re 
IS  far  too  sicplified  to  yield  any  deep  insights  into  the  elements  of 
success  or  failure  in  our  acquisition  process.  I  could  suggest  that  in- 
depth  personal  interviews  of  very  experienced  people  both  in  government 
and  industry  could,  with  effort  (versus  computerized  statistics^  lead  to 
the  insights  that  would  be  valuable  to  AF  mahagement .  There  are  a  great 
nar.y  factors  (political,  personal  characteristics,  role  of  in-heuse  labor¬ 
atories, - etc.)  which  are  often  the  crucial  areas,  as  distinguished  from 

those  simplified  text-book  factors  (adequate  definitior  of  Jot,  work 
breakdown,  management  to  schedule  and  cost,  etc.).  The  subtle  factors  are 
usually  ,rot  openly  discussed,  but  they  are  the  one  that  usually  are  the  key 
to  success  or  failure  of  a  program." 

b.  ."indings  of  Primary  Research. 

(’)  Data  Presentation. 

(a)  Population ■  The  total  respondent  copulation  was, 
'“or  analysis  purposes,  subdivided  into  the  following  groups  based  in  their 
expected  perspective  of  the  system  program  management  process; 


Greup 

Perspective 


IdeTit.!  t'i6r 


lT'.CI 


^ser 

♦ 


-ograc  Cffice 


CGD/FA4E  JG  DSAP,  11.  XJ 
HC  AFsc. ;t  hc  tsaF'R: 


an 


b)  Survey  "esuits.  .'•.IJ  six  if  ‘•.he  c:c..la‘-  pr 


were  represented  by  the  netun".e.d 


1 


Group  Identifier  Number  of  responses 

5 

3 

5 
2 
15 
3 

Data  extracted  from  the  returned  questionnaires  are  pre¬ 
sented  in  the  tables  of  Appendix  3-  Reference  is  made  to  these  tables  in 
subsequent  discussions  of  research  findings: 

Table  I  Classification  of  Success  Criteria 
Table  II  Importance  of  Success  Criteria 
Table  III  Examples  of  Success  Descriptions 
Table  IV  Part  IV  Scores  by  Population  and  Questions 
Table  V  Assumed  Responses  Per  Question  by  Population 
Group 

Table  VI  Average  Response  Per  Question  by  Population 
Group 

(2)  Results  of  survey  to  determine  what  is  meant  by  system 
program  management  success. 

(a)  Evaluation  of  Survey  Questionnaire.  In  evaluating 
the  initial  returned  questionnaires,  it  became  apparent  that  the  Part  I 
instruction  was  not  totally  clear  nor  would  Part  I  be  completely  successful 
in  determining  the  effects  or  attributes  of  successful  system  programs. 
There  were  two  difficulties  encountered: 

2  First,  the  question  "...please  describe  what  you 
believe  is  meant  by  successful  system  program  management"  could  be  inter¬ 
preted  as  either  successful  system-program-management ,  in  which  case  the 
emphasis  was  on  how  effective  and  efficient  the  actual  management  process 
was.  On  the  other  hand,  an  interpretation  of  successful -system  program- 
management  put  the  focus  on  how  well  the  actual  system  turned  out.  This 
dual  interpretation  was  faced  ail  through  this  study:  Does  success  come 
from  the  system  or  the  program? 


A 

B 

C 


F 


( 
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^  Seconc,  ^or.e  r^Jspcnder.ts  :or.cer.tr' 
causes  of  success  r'aCher  CP.ar.  c'ce  af  tri'cuces .  ExaTcles  c: 
cesorintiocs  s.re  : 


"Keeping  "car.agers  aware  cf  prcc  ie-cs  .  .  .  " 
"X’egcc iaPing  cP.e  rigf.c  kinc  c'  ccr.c-acc 
"Tinelv  and  acecuate  funding." 


Parts  II  and  III  of  the  cuestior.na_re  prove  c 
clearer  and  r.ore  succinct  insights  to  what  syste.r.  pregrar.  T.ar.ageter. p  su 
cess  is  perceived  to  oe.  3y  requiring  the  respondent  to  tell  ;^_y  a  pregr 
wns  felt  to  be  successful,  a  '^ather  cosplete  list  of  attributes  was  cc 
3 true tec . 


ho)  .Results,  tables  I,  II,  and  I-I  present  the  consol 
iated  answer  to  what  system  program  aanagemer t  success  means. 

2  Table  I  displays  the  Ilassification  of  Tucce 
Criteria.  The  element  l.I  "Program  and  personnel  rewarded"  was  indue 
based  on  the  secondary  research,  but  no  one,  even  a  respondent  who  had  ne 
awarded  the  Air  Poroe  A.osociaticn  Meritorious  Award  for  Prograh.  Manageme 
or.  two  ocoasicr.s,  li.sted  rewards/ awards  as  ir.cic.at ions  cf  preg^am  ranag 
ment  success.  It  is  important  to  note  that  this  classification  is  n 
exhaustive.  Cost,  for  example,  under  element  -.1.1  "Fulfilled  CC?/? 
Cost.  Schedule/Performance  .CS?''  Pecuirements"  ooulo  have  been  sutdivid 
into  literally  dozer;  of  cu.st  goals.  The  classification  hedudes  t 

L”  cL’ti  3 p. 3. ^ s  u p " 

stand.  This  classi^ica*  ion  of  sucoes.';  criteria  for  a  major  ucquisiti 
program  satisfied  the  Ctucv  Contract  CCVy  requi''ement  I. -.1.4. 


2  Table  Cl  builds  on  ;he  c  .as 
criteria  by  showing,  from  the  /iewpo.nt  of  the  quest 
the  imocrtance  cf  e.acn  t.circ  level  success  c-’ctericn. 


v.oterr.  .vo.: 


ivicuf  rare; 


Element  1  "Accomplished  Program  Objectives," 
though  of  lesser  importance  with  3  "System  Worked  Well  When  Needed,"  re¬ 
vealed  one  interesting  finding.  Respondents  believed  that  a  program  which 
"Maintained  Program  Stability"  was  of  moderate  importance.  Element  1.2 
"Maintained  Program  Stability"  begs  this  question:  Is  program  stability 
an  attribute  (effect)  or  a  cause  of  successful  system  program  management? 
The  importance  of  program  stability  will  be  raised  again  later  in  the 
Section  IV  discussion  of  reasons  for  program  success. 

3  Table  III  lists  success  descriptors  by  criteria 
elements.  The  92  descriptors  were  extracted  from  all  the  returned 
questionnaires.  This  table  is  an  amplification  of  Table  II. 

Appendix  4  to  this  report  contains  several  ques¬ 
tionnaire  Parts  I  which  showed  unusual  perception  of  the  question  of  what 
is  success  in  program  management.  The  reader  is  strongly  encouraged  to 
review  Appendix  4. 

(3)  Results  to  substantiate  or  refute  Hypotheses  I  and  II. 

Hypothesis  I.  System  program  success  is  defined  in  terms 
of  (a)  the  program's  record  in  meeting  directed  cost/schedule/performance 
(CSP)  requirements,  and  (b)  the  program's  system's  ability  to  attain  and 
maintain  a  military  effectiveness. 

Hypothesis  II.  The  closer  one  is  to  the  Program  Office 
(in  an  organizational  sense),  the  more  likely  it  is  that  success  will  be 
defined  in  terms  of  meeting  CSP  requirements. 

(a)  Evaluation  of  Survey  Questionnaire.  The  survey  in¬ 
strument  was  effective  in  meeting  this  research  objective.  Parts  I  and  II 
of  the  questionnaire  provided  data  for  the  first  hypothesis;  this  resulted 
in  the  classification  of  success  criteria.  Part  IV  worked  as  designed  to 
substantiate/refute  Hypothesis  II.  Parts  I  and  II  were  of  little  value  in 
analyzing  Hypothesis  II  because  of  the  broad  range  of  answers  received. 
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Tables 


2  hypctr.esis 
esDonier.bs  blvi 


shown  in 


p^ioheP  Tnogran:  Iblecbives  "  '.r'  "".eebing  birectec;  CS?  require.T.ents"  /  anc 
"Systeni  Aorkeb  'Veil  '*'hen  N'eebec"  ,cr  "systen's  ability  to  attain  and  nain- 
tatn.  a  tilitary  effectiveness"'.  All  descriptors  oculd  be  categorized 
within  ‘hese  two  broad  elertents;  there  were  no  other  categories  of  success 
attrioutes  mentioned. 

£  Hypothesis  II  was  not  substantiated. 

For  analvsis  purposes,  the  factors  which  ray  describe 
systec  program  success  (Part  I'd  of  the  Survey  Questionnaire'  were  ordered 
on  an  assumed  bias  (or  perspective}  favored  by  the  program  office  cr  user. 

Factor ; 


.^rogran 


Perspective 
biased  Toward 


■_isted,  the  factor  ranking  would  be; 


P-cgram  Office  Perspective 


d.  .Veets  directed  decision  Oocrdinating  Paper/Program.  >.an?geT.ent  Oi- 
"ective)  requirements  for  Cost/Sc'redule/Performarce/Suppcrtauility . 

c.  Clearly  defines  an  acquisition  strateizy  at  program's  cutset  and 
ilcsely  adheres  to  it  throughout  the  .ife  of  the  program. 

i.  f.inimizes  design  c'ranges  throughout  development  -.rd  prcduc- 
iicr/deplovmer.t . 


f.  defines  an 


phase 


h.  Incorporates  the  most  current  state-of-the-art  technology  in  the 
system  design. 

e.  Produces  a  system  for  which  the  operational  requirement  has  been 
clearly  established. 

b.  Integrates  support  and  manpower  concerns  into  the  acquisition  pro¬ 
cess  such  that  system  deployment  is  accomplished  with  minimum  disruption 
to  the  operational  force. 

j.  Meets  an  established  Initial  Operational  Capability  (IOC)  date. 

a.  Minimizes  the  time  from  need  identification  to  introduction  into 
operational  use. 

g.  Deploys  a  system  with  capability  which  is  competitively  effective 
against  the  threat  during  the  system's  projected  life. 

( _  User  Perspective 


In  Table  IV,  (App.  3)  the  Part  IV  Questions  (i.e.,  fac¬ 
tors)  are  arranged  in  order  of  expected  perception,  Program  Office  on  left. 
User  on  right.  The  population  groups  are  placed  with  Program  Office 
orientation  at  the  bottom,  then  User  orientation  at  the  top.  Actual 
responses  to  the  questionnaire  Parts  IV  were  then  listed. 


Table  V  (App,  3)  displays  an  expected  assumed  response 
per  question  by  population  group  if  Hypothesis  II  was  valid.  A  graphic 
description  of  this  would  be: 


User 

f 


uroups 
Orientation 


10  1  How  answers 

should  be  if 
Hypothesis  II 

1  10  were  valid. 


Program 

Office 


Question  Perspective 
Bias  Toward 


User 


Table  VI  (App.  3)  attempts  to  remove  the  heavy  Air  -oroe 
3PD  orientation  by  averaging  the  Table  I  responses  by  population  group. 
Here  are  some  data  results. 

Relative  importance  of  each  question  (factor)  irrespec¬ 
tive  of  population  group: 

"actor  dcifhebjag 

Average  of 
^^rouo 

average  4.27  4.57  7.3  6.25  7.93  3.97  4.93  7.83  5.33  5.' 

responses 

Indicated 

ranking  3  9  8  710  2  5  96  1 

Data  from  the  Parts  IV  were  analyzed  and  show  that  Hypo¬ 
thesis  II  is  refuted. 

Annex  1  to  Appendix  3  displays  the  statistical  analysis 
to  refute  Hypothesis  II.  If  Hypothesis  II  were  valid,  a  distribution  of 
Part  IV  responses  would  be  statistically  equivalent  to  Table  Y.  In  Annex 
1,  the  Null  Hypothesis  is  that  the  means  for  groups  A,  C,  and  E  and 
responses  for  factors  d,  h,  and  g  are  as  shown  in  Table  V.  The  statistical 
test  shows  conclusively  that  the  assumed  means  are  not  within  the  true 
means . 


Results  from  Annex  2  to  Appendix  3  suggest  that  an  alter¬ 
nate  Hypothesis  II  may  be  correct:  System  program  management  success 
tends  to  be  defined  by  similar  terms,  irrespective  of  organizational  per¬ 
spective.  In  this  statistical  analysis,  an  Analysis  of  Variances  for  Ine- 
Way  Classification  at  a  5?  significance  level  showed  no  reason  to  reject 
the  hypothesis  that  the  average  response  of  factor  "d"  from  Part  IV  was  the 
same  for  every  group.  If  further  analysis  verified  the  Altcnat  .  .-e  Hyro- 


OC 


thesis  II  for  all  10  factors,  the  data  would  show  that  the  two  most 
important  factors  from  Part  IV  to  describe  success  are: 

Factor  g.  Deploys  a  system  with  capability  which  is 
competitively  effective  against  the  threat  during  the  system's  projected 
life. 


Factor  e.  Produces  a  system  for  which  the  opera¬ 
tional  requirement  has  been  clearly  established;  further, 


are; 


the  data  would  show  that  the  two  least  important  factors 


Factor  h.  Incorporates  the  most  current  state-of- 
the-art  technology  in  the  system  design. 

Factor  j.  Meets  an  established  Initial  Operational 

Capability  (IOC)  date. 

C.  Observations  Concerning  Success  Criteria. 

This  subsection  generalizes  the  findings  of  Section  III,  At¬ 
tributes  of  Success.  Appendix  3,  Table  I  "Classification  of  Success  Cri¬ 
teria"  lists  all  attributes  of  success  suggested  by  the  research  study. 
Based  on  an  analysis  of  all  the  data,  these  observations  are  offered: 

1 .  Program  management  success  is  generally  determined  by 
product  success,  in  the  non-defense  and  defense  (military)  sectors. 

2.  Attention  to  system  performance  and  system  supportability 
is  more  important  to  long-lasting  success  than  is  attention  to  program  cost 
and  schedule. 


3.  A  program  whose  system  is  usable  and  meets  a  need  (whe¬ 
ther  or  not  the  need  was  known  at  program  initiation)  will  be  considered 
more  successful  than  one  which  meets  unusable  or  unnecessary  performance 


requirements.  This  observation  can  be  carried  one  step  furtner  to  state 
that  system  program  management  success  is  more  of  a  function  of  what  impact 
the  system  had  on  the  Air  Force's  military  capability  tear,  cn  how  well  the 
technical  performance  goals  were  met.  One  respondent  suemarited  this 
important  point  with  the  statement  that  a  "Useful  product  is  the  savior  of 
any  program." 


SECTION  IV  CAUSES  OF  SUCCESS 


Overview.  This  section  responds  to  Section  C.4.2  of  the  SOW.  There  is  no 
attempt  to  provide  a  functional  "How  To"  check  list  on  program  management. 
The  order  of  presentation  is;  a  review  of  a  broad  classification  of  suc¬ 
cess  causes,  a  discussion  of  successful  Air  Force  system  programs,  a  sug¬ 
gested  list  of  the  most  important  inputs  (causes)  to  enhance  system  program 
management  success,  comments  concerning  Hypotheses  III  and  IV,  and  some 
overall  observations  concerning  success  causes. 

A.  Classification  of  Causes  for  Success 


Figure  IV-1  presents  the  top  level  classification  causes  for  system 

1  ■ 

program  success.  This  classification  was  developed  from  secondary  re-  i 

search,  experience  of  the  study  team,  and  results  from  the  study  question- 
naires  and  investigative  worksheets.  Examples  of  each  of  the  subelements 
are  included  in  Table  IV-1  located  directly  behind  Figure  IV-i.  Every 
questionnaire  and  investigative  worksheet  example  of  success  cause  is  cov¬ 
ered  by  an  element  in  the  table. 


(NOTE:  Section 


IV-B  begins  following  Table 


;v-i ). 
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Figure  IV-I  Clasnlf  )<  at  Ion  uf  Caiiaes 


TABLE  IV-'.  EXAMPLES  OF  OESCRIPTCKS 
FOR  CAL'SES  OR  REASONS  FOR 
SYSTEM  RROGRAM  SL’CCESS 


'.0  Attention  to  Directed  Coat/Schedule/Performance  OS?'  Sequ: resent:  s . 

1.'  Executing  directed  prograir.  within  the  resources  provicec. 

A.  Bringing  system  into  inventory,  on  schedule,  witr.qr.  nest 
estimates,  ana  meeting  specif. oatton3--the  t.ass.ca. 
definition . 

B.  Meeting  the  requirements  of  cost,  schedule,  and  perfor¬ 
mance  is  the  objective. 

C.  Living  up  to  promises  made  to  OSD  through  the  DC?  process 
and  to  the  Congress  through  the  SAP  process. 

D.  Meeting  cost-effective  (not  "Letter  of  the  Spec")  perfor¬ 
mance  requirements. 

’.2  Meeting  CSP  requirements  or  cancelling  program. 

A.  If  product  cannot  meet  all  CSP  goals  and  objectives 

"(they  are  never  requirements),”  either  stop  program  or 
sensibly  relax  goals  until  a  useful  product  can  be  real¬ 
ized. 

3.  Program  is  cancelled  if  it  becomes  clear  that  the  costs 
exceed  the  benefits. 

C.  Do  not  drag  a  program  out.  Develop  in  the  shortest  time 
possible  or  cancel. 

D.  Operational  performance  objectives  demonstrated  during 
test /evaluation . 


'.3  Strict  adherence  to  CSP  requirements  in  following; 

.j.’i  Adherence  to  system  performance. 

'.3.1  Aaherence  to  system  supportability . 

■.3.3  Adherence  to  program  schedule. 

■.i."*  Adherence  to  program  cost. 

1.**  Ability  and  initiative  to  perform  meaningful  trade-offs  of  CSP 
requirements . 

A.  Ability  to  balance  resources  against  significant  underly¬ 
ing  requirements  while  neutralizing  the  impact  of  trivial 
requirements . 

5.  Knowing  which  of  the  CSP  requirements  really  is  the  most 
important  and  making  trade-offs  accordingly. 

2.0  Ability  and  success  in  maintaining  tight  control. 

2. I  Overall  program  control. 

2.1.1  Organizational  management. 

A.  Comprehensive  and  dynamic  planning. 

B.  Finn  baselines  with  traceability. 

C.  Risk  management  is  understood  and  performed. 

D.  When  problems  involving  cost,  schedule  and  quality 
surface,  they  are  acted  on  promptly. 

E.  The  program  manager  (PM)  is  decisive. 

F.  Getting  all  functional  areas  to  work  together  as  a 
team. 

G.  The  PM  does  not  get  involved  in  the  day-to-day  de¬ 
tails. 

H.  Contractor  must  have  a  good  management  system. 


(TABLE  IV- 1  Continued) 
42 


T.  '.^ntracto^  is  irlver.  au*hori‘v  and 


arnu -Si •■  \ '.’n  s”. wr.  .c 


if*c^ao  riann'.'v’  .and  i'jdg>=t.  ing . 


a:  .'.  snaer.i  •'  -.a-p.:'»>. 


N.  nn  ao 


jr-  p--a’>»gv  da.sei  sr  '  •  <’  •  •a., 


i.'..,  -harve  T.ar.aflr«n;er;t  . 


A.  Adi..;','/  :  'cri"^.  .'r.anves. 

r .  rirfeotive  ccr'raci  onar.^e  sv>i^r. 

yaindain  fi'"d  neq nemer."  s  .  ,  ;i 

in»=‘n  -.n  '  .^'  i . 


Sesod’"''-’  'ortncl. 


r^chnical  ccr.trol. 


A.  Translation  of  user's  cperai .  jr'.a '.  sr-  . 

to  design  criteria  fui:,.'/  w.Zh:-.  ita'd"-,  •'- 
5.  Correc*-  early  definition  i:  i.-.e  ‘  •'■-"ir  .  - 

C.  Carefully  define  me  .sa^iei  * '> 

D.  Cesigning  a  syster.  -ee-  mn  cena'  .  ■ 
xent  without  gel d-i i  a*  mg 

S.  Avoiding  ‘echr.i’al  -^xie  ,, -.srisen^  ■ . 

?.  Con't  increase  perf  arxan.  e  :ey:r.;.  s*.!*--’- 
G.  Sarlv  derini-.:or,  ?f  *he  'nehn-os'.  s 


'A?GZ  .V-- 


c  Cost  Control 


A.  Negotiating  a  contract  that  protects  the  taxpayer's 

\ 

interest .  • ;  _ 

3.  Develapr.ent  ^ realist..:  life  cycle  cost  data. 

C.  Properly  using  .i:'e  cyc.e  cost  as  a  guicing  factor. 

2.  Cost  anc  scnedule  relationships  estaolisned. 

3.  Accurate  cost  estiiates. 

Scnedule  centre.. 

A.  Development  cf  a  realistic  schedule. 

5.  fcllcwing  a  consistent,  well-developed  set  of  mile¬ 
stones  . 

C.  Meeting  the  schedule  milestones, 
ed'uacy  of  means  to  accomplish  program  oojectives. 


.Resource  aval  Laoi.ity . 


Personnel  availaPiiity. 


A.  Se.ect  highly  cjualified  people  to  manage  the  program. 
5.  Program  support  provided  either  through  project  or 
matrix  organisation . 

C.  Personnel  continuity  maintained. 

D.  CooG  experienced  people  put  in  onarge. 

3.  Icod  SfC  team. 


7  e  c  n  n  c  1  o  g  y  a  v  a  1 1  a  1 1 1 1 1  y  . 
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Z.  Using  available  technology. 

1.  Testing  cosponents  prior  to  project  tevf^  ctTe'C:  . 
E.  Scall  step  forwarc. 


3.  -3  Financial  availability. 


A.  i^uncling  is  consistent. 

3.  Assure  adecuate  F'cndlng  for  work  to  Pe  perforred. 

Tir.e  availabiltt'-' . 

Realistic  schedules  wh;ch  are  compatible  with  the  system 
requirements . 

3.2  Effective  organisation. 

A.  The  SPO  demonstrates  excellence. 

5.  Establishment  of  a  management  team  with  responsibilities. 
C.  Selecting  proficient  contractors  wit.h  recent  similar  ex¬ 
perience  . 

T.  Selecting  responsible  contractor  with  best  chance  o'" 


neetlr,g  program  requirements. 

E.  The  contractor  demonstrates  excellence. 

?.  Good  contractor  management. 

G.  Aggressive  management  engagement  with  contractor. 

H.  User  is  integral  part  of  SPO. 


Support  for  the  system  program. 


Support  above  the  program  office  level. 


3e  sure  system  has  .adequate  support  at  the  “op  in  “  "s  trier 
importance ; 


HQ  USAF 
OSD 

Congress 


-».2  Support  at  working  level  or'  the  program  office. 

A.  Continued  management  commitment  by  government  and  indus¬ 
try  over  life  span  to  develop,  produce,  and  deploy  the 
system. 

B.  Disciplined  participative  management  between  developer, 
logistics  manager,  user,  contractor,  and  headquarters. 

C.  Interaction  with  training,  logistics  and  operational  com¬ 
mands,  and  industry  so  that  when  the  system  becomes  oper¬ 
ational  it  fulfills  a  significant  role. 

4.3  Actions  by  the  program  manager  (PM). 

A.  PM  is  aggressive  in  selling  and  explaining  the  program 
and  progress. 

B.  ?M  is  straightforward  in  program  assessment  and  acknowl¬ 
edges  problems. 

C.  Diplomacy. 

D.  Keep  program  visibility  low. 

5.0  Existence  of  clear  requirements. 

5.1  Clarity  and  understanding  of  the  requirement. 

A.  Clearly  stated,  fu^ly  justified  and  realistically  attain- 
aole  operational  specifications  that  accurately  relate 
the  user's  -.eeds  for  fulfilling  his  role  in  supporting 
national  objectives. 

5.  Clearly  articulated  mission  requirements.  They  are 

readily  understood. 


f 
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C.  The  PM  is  sure  there  is  a  need  for  the  system. 

T.  Pealistic  and  technically  supportable  requirements. 

E.  The  time  (need  date)  requirement  for  the  system  is  appar¬ 

ent  . 

P.  Pequirements  well  defined. 

5.2  Forcefulness  of  the  requirement  statement. 

A.  The  need  is  recognized  by  the  organizations  whose  support 
is  necessarv. 

3.  The  requirements  are  firm  and  quantified. 

C.  The  ROC/SONS  is  validated. 

D.  The  MENS  is  approved. 

S.  Meeting  the  requirement  will  complement  ether  system's 

effectiveness . 

5.3  Stability  of  requirements. 

A.  Stable  program. 

3.  Make  sure  user  knows  what's  neecec  at  the  teeir.r. tng. 


3 .  Deternination  of  Successful  Programs 

To  determine  which  Air  Force  system  programs  were  to  be  ranked  in  the 
top  success  classes,  the  following  inputs  were  required: 

v1)  List  of  all  USAF  major  programs  which  have  experienced  a  DSARC 
III  (or  equivalent)  milestone  from  1  July  1965  through  30  April  1980. 
Taole  IV-2  separates  the  40  major  programs  into  22  to  be  evaluated;  and  16 
to  not  be  evaluated  with  reasons. 

(2)  Investigative  worksheet  (See  Appendix  5)  with  elements  of  suc¬ 
cess  criteria  and  input  parameters. 

(3)  Results  from  the  Phase  II  investigation  of  all  programs  on  the 
population  list.  These  results,  listed  in  Appendix  6,  represent  the  study 
investigator's  subjective  evaluation  of  each  program's  strength  in  meeting 
the  attributes  of  success  and  of  demonstrating  causes  of  success. 

Based  on  the  above  inputs,  the  ranking  in  Table  IV-3  represents  the 
findings  of  the  study  team.  Qualification  statements  beside  each  program 
amplify  reasons  for  its  rating  in  the  "Most"  or  "Moderately  Successful" 
Cx3.SS63  • 
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TABLE  IV-2  PROGRAM  POPULATION 


ProRrams  To  Be  Included 


U  A-7 

2.  A-’.O 

3.  Air  Force  Satellite  Communications  System  (AFSATCOM) 

.  Air  Launched  Cruise  Missile  (ALCM) 

5.  B-l  (Advanced  Manned  Strategic  Aircraft,  AMSA) 

6.  C-5A 
T  .  C-  '■  L 1 

8.  DOD  Space  Transportation  System 

9.  Defense  Satellite  Communications  Systems  (DSCS)  I  and  II 
"O.  E-3A  (Airborne  Warning  and  Control  System,  AWACS) 

'  1 .  E-^  (Advanced  Airborne  National  Command  Post,  AABNCP) 

*2.  EF-111A 

F-5  E/F  (International  Fighter  Aircraft,  IFA) 

•4.  F-15 

•5.  F-:6 

'6.  F/RF/FB-111 

17.  Gunship 

18.  KC-lOA  (Advanced  Tanker/Cargo  Aircraft,  ATCA) 

'9.  Maverick 

2C.  Minuteman  II  and  III 

21.  NAVSTAR  Global  Positioning  System 

22.  Short  Range  Attack  Missile  (SRAM) 


♦ 


Programs  Net  To  3e  Incluaed 


Name 

1 .  Aavanced  Ballistic  Re-£ntr>  System 
(ABRES) 

2.  ATC  Radar  Beacon  IFF  Mark  XII 
System  (AIMS) 

3.  Aavanced  Medium  SIOL  Transport 
(AMST; 

4.  Aero-Propulsion  Systems  Test 
Facility  (ASTF) 

5.  C-141  Stretch 

6 .  DOD  COMSAT 

7.  Drone/Remotely  Piloted  Vehicle 

8.  Defense  Support  Program  (DSP) 

9.  f/RF-4 

10.  Ground  Launched  Cruise  Missile 
((GLCM) 

1 1 .  Pave  Strike 

12.  Precision  Action  Link  (PAL) 

13.  Precision  Location  Strike  System 
(PLSS) 

14.  Program  647  (CLASSIFIED) 

18.  Saores  (Self  Aligning  Boost 
Re-Entry) 

16.  Space  Defense  System  (SDS) 


Rationale 

A  "conglomerate"  program  of 
continuing  cevelopments/ improvements 
of  subsystems. 

Not  a  system  but  a  family  of 
equipment . 

Program  cancelled  prior  to 
DSARC  II. 

Not  a  defense  system,  but  more 
of  a  real  estate  project. 

Really  a  modification  program. 

Now  called  DSCS. 

Not  one  system,  but  many  programs. 

Not  one  defense  system. 

First  production  contract  (F-4C) 
signed  in  1962. 

Not  in  Production/Deployment. 

Accelerated  development /product ion 
of  selected  systems  (not  one  system 
program) . 

Too  old;  essentially  a  space 
technology  program. 

DSARC  III  scheduled  1985. 


No  data  available. 

Too  old;  a  space  technology  program. 
Too  class.Lfiec;  .not  deployed. 
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17. 

Simulators 

Not  one  system,  but  many  separa' 
programs . 

18. 

Titan  III 

A  program  to  develop/produce 
versatile  boosters  for  space 
exploration  purposes.  Titan  II 

the  weapon  system,  was  in 
production  before  1365. 

Succ.jsi 
Cr: teria 

froKrax  Score  Coxr.er.ts 


C-'.-l  -i.c  0  Co.T.S.nec  !'-A.S'Ar„C  teaoir.g  was  owtscar.dir.g. 

0  V.ec.'.a.x  war  causes  -rger.t  reqairerr.er.t . 

0  Significant  otc.c  forward  an  capaoility  ar.i  teohnclcgy. 


Maverick  )).'•*  o  Great  military  utility. 

o  user  knew  .now  to  use  systex  tecause  of  extensive  testi.ng. 
0  Vietnam  a.na  Mic-east  wars  oausec  urgency. 


r-5  £/r  0.0  o  Hig!-.  reliaoility  anc  adaptaoi lity . 

c  Succeeded  cespite  lacx  of  ISAf  interest. 


TABLE  IV-3 


PROGRAMS  BY  SUCCESS  CATEGORIES 


Yost  Successful 

Least 

Successful 

Program 

No . 

Responses 

Program 

N'c  .  !=espcr.ses 

:-15 

1  :■ 

C-5A 

2 1 

C-'.4  1 

F/RF/FB-1 1  ’ 

-6 

F-'6 

8 

3-1 

MM  II  4  III 

b 

DOD  STS 

*4 

E-5E/F 

PLSS 

4 

Maverick 

b 

Drone/RPV 

3 

Cunchip 

b 

E-4 

j 

A-’'C 

5 

AMST 

F/RF-M 

c 

EF-1 1 1A 

c 

A-7 

4 

F-15 

i- 

5-1 

U 

GLCM 

2 

Titan  III 

SDS 

2 

E-3A 

2 

Simulators 

•*> 

c. 

.VAVSTAR  CPS 

4. 

AFSATCOY 

*! 

SRAM 

2 

ALCM 

1 

A3RES 

1 

KC-’OA 

• 

AFSATCOM 

MM  i: 

ALCM 

1 

Titan  III 

1 

AMST 

I 

DS? 

5-4 

EF-n  u 

F/RF/FB-1 1 1 

1 

:<c-iOA 

1 

Frog  6L7 


TABLE  :V-‘4  MJ..JCR  FRCGRAYS  FELT  YOST  AS'D  LEAST  SUCCESSFUL 


C .  Comparison  of  Successful  Program  Rankings 

1 .  Requirement 

Study  Task  II-E  required  that  phe  programs  in  the  top  two  Success 
Classes  be  compared  with  the  "list  of  programs  felt  to  be  successful"  from 
Task  I-E.  Significant  anomalies  were  to  be  identified  and  discussed. 

2.  Criteria  and  Assumptions 

For  this  comparison,  the  following  assumptions  and  criteria  were  used 
to  identify  "anomalies": 

a.  The  program  ranking  obtained  through  application  of  the  suc¬ 
cess  criteria  (i.e.,  Table  IV-3)  is  considered  the  baseline  for 
comparison. 

b.  Programs  rated  in  the  top  11  of  baseline  ranking,  but  not  in 
the  top  11  of  major  programs  felt  most  successful  (see  Table  IV-4,  were 
considered  anomalies. 

c.  Programs  mentioned  three  or  more  times  as  "most  successful" 
i^Tatle  IV-4)  but  not  included  in  the  top  two  classes  through  use  of  success 
criteria  were  considered  anomalies. 

3.  Discussion  of  "Anomalies" 


a.  High  in  Ranked  Success  But  Low  in  Perceived  Success 

(1)  E-3A.  It  can  be  speculated  that  this  succtssfu*  prugri: 

was  underrated  because  of  its  relacive  newness  and  complexity  of  miss.cr, . 
Also,  though  very  effective,  the  E-3A  cannot  be  generally  regarcec 
"exciting. " 


V 


This  prcgras  suffered  early  adverse  publicity,  and 
eployed  a  weapon  system  which,  ssoueh  very  effective  as  a  deterren*' , 
eceived  lictle  publicity. 

'  ■  KC-'CA.  This  r'-jsfvap  l.c  very  new,  wish  the  system  yet  to 

e  deployed,  lue  to  the  very  nature  of  the  orograc,  "espondents  may  have 
uhSiv  isir.iuiurc  *■. 00’^.^. l6ng0'  r0!rov0(j  I’luoh  progrp.T  rT’&r.a§0ni6nt 

b.  Felt  to  be  High  Though'  los  Tanked  High  Using  Success  Criteria 

Titan  I'l.  Respondents  oos.eibly  were  rating  the  entire 
'icar.  ; 1 ,  11,  and  111'  .'smily  which  ieployed  a  key  element  in  the  United 
states'  nuclear  deterrence.  For  this  study,  however,  the  Titan  III  was  not 
'Oted  using  the  success  cniter'.a  h'-''cau.re  of  Titan  Ill’s  main  use  as  a  space 
n.cste-. 

;  1 3-'.  This  prigran  preser.tec  one  of  the  cleanest  examples 
*  hx"  hi ‘'fe''®'"ces  between  program  manageiaent  and  sy.stem  program  manage- 
•er:  -:v . cer.cc  exists  snowing  the  excellence  and  acccmplishments 

'  ■■■'.•-'uh  direct, -r  and  his  rta"'*'  because  of  results  in  terms  of 

:a  ,  i .;h .-’v^^m'^nt  an'  ■“eal  cost  control,  'he  progr.aic.  like  the 


ho  usable  systems  were  deployed 


1 


terms  of  military  effectiveness,  no  other  Air  Force  weapon  system  has  ever 
had  more  importance  than  Minuteman  because  of  its  deterrent  value. 
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r  a 

ctors  Dcr.tr  touting  tc  o^coess 
roi  Duest lor.naire  rant  V 

f  re-:;uer.cy 
of  Response 

.'‘.6- at.  1  Vtr 

Ircpor  tar.c 

.  3 .  . : 

a . 

ROC/SCNS  Validatec 

ii':’ 

Some 

►- 

DCP  Aiternative  Defercec 

-OW 

r  ,  -  *  r^r.er.T 

c  • 

MENS  Acceptec 

"■ 

—  C  ’4 

Responsiveness  to  Threat 

-  i 

nigr. 

e . 

Priority  Time  Urgency 

‘•L- 

Mocerate 

c 

Complement  to  Other  Systems 

33 

Some 

.  -  .  w  rr  t 

a. 

Congress 

-0 

Mocerate 

:  *’ 

d. 

OSD 

■Mocerate 

o  y  i  t,  c  :a 

c  • 

HQ  USAF 

C  ** 

nigr. 

.- .-’  33 ; 

A 

• 

USN  or  US  Army 

•  ; 

—  c  w 

e. 

Political  Pressure 

2*^ 

Seme 

f . 

Unity  of  Effort  ;AF,  CSD,  Cong} 

40 

Moderate 

'■leans 

a. 

Funding  was  Consistent 

73 

High 

X.  0 

b. 

Contractor  Demonstrated 

Aoocmplish 

Excellence 

_  ^ 

High 

c. 

SPO  Demon.strated  Excellence 

59 

Mocerate 

d. 

PEM  Excellence 

2C 

uOW 

Jniical 

a. 

OSD 

45 

Mocerate 

*\"cgra3i 

b. 

HQ  USAF 

Moderate 

Decision/ Actions 

c. 

HQ  AFSC  or  Dev.  Center 

30 

Some 

Dutside  S?0 

d. 

CMD 

\  \ 

Low 

w  t  n  c  t 

a. 

Prograim  Cost 

50 

Mocerate 

Adherence 

b. 

Program  Schedule 

^  . 

.Mocerate 

to; 

c. 

System  Performance 

7  4 

Highest 

d. 

System  Supportability 

5^ 

Moderate 

Dt.'.er 

a. 

High  Security  Classification 

4 

:.ow 

0. 

Application  of  New  Technology 

20 

Low 

G  . 

Other : 

\  ■. )  Wartime  exigency 

2 

1.0W 

\Z',  Reasonable  goals 

\3'  .Multinational  Involvement 

2. 

Lew 

>>■«)  Variety  of  Use 

Low 

30 


(3)  The  system  was  supportec  by  HT  USAF. 

(4)  The  requirement  was  responsive  to  the  threat. 

(5)  The  contractor  demonstrated  excellence. 

b.  From  Investigative  Worksheet  results,  Table  IV--c; 

( 1 )  Availability  of  Technology- 

(2)  Adherence  to  system  performance  requirements. 

(3)  Adherence  to  system  supportability  requirements. 

(‘i)  Continuity  and  authority  of  ?M. 

(5)  Emphasis  on  change  management. 

(6)  Availability  of  program  office  personnel. 

(7)  Availability  of  time  (for  development). 

0.  From  a  correlation  analysis,  output  success  scores  (Table  IV- 
3)  with  input  parameter  averages  (Appendix  6): 

( 1 )  Input  parameter  2  "Ability  and  success  i.t  maintaining 
tig.^t  control,"  r  :  .76. 

(2)  Input  parameter  4  "Support  for  the  system  program," 

r  -  .04. 

(3)  Input  parameter  1  "Attention  to  directed  CS?  require¬ 
ments,"  r  -  .57. 

(4/  Input  parameter  3  "Adequacy  of  raea.ns  to  accomplish  pro¬ 
gram  cblectives,"  r  =  .42. 

(5)  Input  parameter  5  "Existence  of  clear  requirement," 

r  -  .  )  ’ . 

wVOTE:  Analysis  showed  there  was  no  satisfactory  multiple 
■-inear  regression  expression  which  would  define  the  relationship  oetween 
t.-e  -'’ive  input  parameters  and  results  of  the  eleven  top  programs.' 

F .  Hypotheses  III  and  IV 

Hypothesis  III:  System  program  success  is  impeded  because  the 
Program.  Manager  is  disciplined  to  meet  Cost/Schedule/Perf ormar cs  (CS?. 


viauals  indicate  that  system  (or  product)  success  overshadows  adherence  to 
CSP  requirements,  and  if  system's  effectiveness  results  require  that  the 
technical  performance  baseline  be  changed,  then  the  SPD  is  obligated  to 
work  for  such  a  change.  Program  stability  is  desired,  but  no  evidence 
shows  tnat  stability  is  more  important  than  expected  system  performance. 

2.  Hypothesis  IV:  The  most  important  determinant  of  system  program 
success  is  the  clarity  and  understanding  of  the  Intended  military  mission. 

a.  Discussion.  The  analysis  method  to  substantiate  this  hypothe¬ 
sis  was  to  consider  each  system  program  which  was  included  in  the  two  top 
success  classes  (Table  IV-3).  If  each  successful  program  had  a  clear  and 
easily  understood  military  mission,  the  hypothesis  would  be  supported. 


?r' 

ogram 

Easily 

Understood 

Military 

Mission 

Selected 

Comments 

1 

.  C-141 

Yes 

"More  truck  than  limousine,  more  mule  than  race 

horse." 

2 

Maverick 

Yes 

Kills  tanks. 

3 

F-5E/F 

Yes 

Support  Nixon  doctrine. 

?-  iD 

Yes 

Low-mix  air-to-air. 

5 

E-3A 

No 

Effective  system  for  complex  mission. 

0 

Gunship 

Yes 

Hamlet  defense  and  truck  killer. 

7 

F-15 

Yes 

World's  best  air-to-air. 

6 

SRAM 

Yes 

Assist  bomber  penetrator. 

9 

A-7 

Yes 

Close  air  support. 

To 

A- 10 

Yes 

Close  air  support  to  kill  tanks. 

'  ■ 

KC-10 

No 

Tanker,  cargo,  or  ??? 

Note  that  two  of  the  programs  do  not  fit  the  criterion  of  easily 
understood  military  mission.  Additional  evidence  tending  to  refute  Hypo¬ 
thesis  IV  comes  from  the  investigative  worksheet  results  (see  Table  IV- 
o):  "Availability  of  Technology"  seemed  to  be  the  most  important  input 
sub-parameter . 


b. 

judgement 


Judgement.  Hypothesis  IV  cannot 
is  based  on  qualitative  analysis 


be  substantiated.  Ihir 
."ather  than  statist  ica. 


results . 


.  Cvenall  Observations  Concerning  Causes  of  Success. 

"nom  the  sub-section  rJ-l(above;  sets  of  success  causes,  it  is  diffi¬ 
cult  to  substantiate  the  existence  of  a  definitive  ranking  of  success 
causes.  There  are,  .lowever,  reasons  to  believe  t.nat  t.tese  observations  are 

valid; 

i.  Availability  of  technology,  and  strict  adherence  to  system  perfor¬ 
mance  should  be  considered  the  highest  priority  causes.  As  evide.nce  for 
this  observation,  consider  the  methods  used  by  t.hese  successful  programs 
to  assure  that  the  needed  technology  was  in  hand  and  that  the  user  under¬ 
stood  the  expected  military  utility  of  the  system: 

a.  Extensive  operator  testing  concurrent  with  early  prc- 
duGtion/deployment .  System  shortcomings  were  detected  and  corrected. 

b.  Maverick:  Extensive  systems  testing  with  operator  involve¬ 
ment. 

0.  E-5E/F; 


(  ;  '  Follow-on  from  a  mature  .'■ystem. 


'.£)  Prototype  demonstrated  technology. 


d.  F-'o:  Prototype  cemcnstrated  technology. 


e.  E-3: 


[  'i )  Extensive  involvement  in  prototype  testing  showed  opera¬ 
tor  what  the  system  could  do. 

(2)  Selected  sub-systems  were  well-proven  prior  to  integra¬ 
tion. 

f.  Gunship: 

( 1 )  Extensive  operator  involvement  during  prototype  evalua¬ 
tion  snowed  tremendous  utility  of  system. 

(2/  Relied  mostly  on  off-the-shelf  components. 

g.  A-7 :  Follow-on  from  a  mature  system. 

h.  rt-10: 

(1)  Prototype  demonstrated  technology. 

(2)  Only  after  consiuerable  operator  testing  did  operational 
concept  and  utility  evolve. 

i.  KC-10:  Modification  of  a  mature  system. 

2.  Input  parameter  5.0,  "Existence  of  clear  requirements,"  appears  to 
be  the  least  reliable  indication  of  program  success.  Historical  data  shows 
that  four  of  the  successful  programs  were  virtually  forced  upon  the  Air 
Force:  F-3E/F,  F-l6,  A-7,  and  A-lC.  Even  with  the  apparent  success  of 
these  programs,  the  Air  Force  currently  does  not  have  a  clear  use  for  the 
F-5E/F  (other  than  a  specialized  "aggressor  training"  role)  nor  the  A-7. 

Certain  of  the  programs  have  appeared  more  successful  tnan  others 
oecause  urgent  requirements  arose  which  tney  could  meet  ve.g.,  in  tne  CEA 
conflict:  A-7,  and  Gunship;  anc  in  the  Middle  East  conflicts: 


Maverick,  E-3A).  Hence,  the  conditions  of  history  sometiise  allow  certain 
systems  to  excel,  while  others  (Minuteman  and  SRAM)  are  effective  in  a 
totally  opposite,  yet  unspectacular  manner:  not  having  to  be  vsel  hecavst 
of  their  succesj  as  deterrents. 

3.  System  success  (hence  program  success)  tends  to  be  teasurec  in 
terms  of  how  well  the  following  seemingly  straightforward  luestltr.o  can  te 
answered  in  the  affirmative: 

a.  Does  the  user  understand  what  to  do  with  the  system  to  obtain 
military  utility? 

b.  Does  the  system  work  well? 

Included  within  these  questions  are  the  considerations  of  operational 
concept,  doctrine,  and  military  utility;  as  well  as  capability,  reliabili¬ 
ty,  and  adaptability. 

A  credible  case  could  be  made  that  the  demise  of  the  A.MST,  S-',  and  5- 
?D  programs  resulted  in  a  large  part  because  these  two  questions  could  not 
be  answered  satisfactorily.  If  the  user  can  perceive  a  definite  and 
important  use  for  the  system,  the  chances  for  program  completion  are  high, 
Without  this  clear-cut  understanding  of  what  significant  nole  the  system 
will  fill,  the  orogram  can  expect  failure.  Just  as  the  3-7C  failed:  "The 
ultimate  cause  cf  failure,  shared  by  two  admanistrations ,  was  t.'iat  the  H-~l 
could  not  do  a  useful  fcb."  ('p,  ;-9', . 
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SECTION  V  SOME  IMPACTS  OF  FINDI'JGS  OV 
AIR  EO^^CE  SYSTEV  PROGRAM  MAN'Av~!-EME!JT 

A.  Inr:roductlor .  The  flncings  of  this  study  suggested  eertair.  "ssues 
which  the  study  teaa,  based  on  Its  experience  and  judgement,  believe; 
should  be  the  subjects  for  further  investigation.  Each  topic  involves  an 
aspect  of  organisational  ef ‘'ectiveness  which  helps  to  deternine  the  over¬ 
all  success  of  Air  Force  systen:  propran;  management.  "^he  ciscussicn  of 
these  issues  becomes  the  foundation  for  the  Recommendations  for  Additional 
Research  found  later  in  this  report  (see  page  ""  . 

3.  Issue  it'.  Importance  of  Military  Utility. 

Question .  Assuming  that  the  ability  of  a  system  to  perform  a  signifi¬ 
cant  military  mission  is  one  of  the  most  important  determinants  cf  system 
program  management  success,  what  methods  are  used  within  the  program  of¬ 
fice  and  by  the  acquisition  management  review  process  to  estimate, 
measure,  and  control  military  utility? 

discussion.  The  concept  of  the  Mission  Area  Analysis  'MAA,>  is  a 
method  to  relate  system  performance  to  mission  effectiveness.  MJiA  outpu' 
can  either  lead  to  an  app"oved  "Mission  Element  Need  Statemort  'MENSi  t.- 
serve  as  an  annual  revalidation  of  the  system's  eventual  worth.  The  MA,> 
process  misses  the  point  of  this  'ssue,  however.  Wlt.hir  the  troject  offiie 
and  acquisition  review  p-ocers  there  is  a  need  for  a  method  to  relate  the 
system's  planned  mllita’"y  utility  to  the  prot'ram's  ccst/schedule/per fcr- 
mance/suppo"tability  (I/S/?/S(  requirements .  Seme  technique  employing  a 
capability/threat  ratio  may  b-"'  apo^'^priate.  T'"°  imoortart  joint  is  tr.a‘ 
the  program  manager  must  hav^  ':r.e  ability  to  "“late  and  demonstrate  t",- 
impact  of  C/.S/P/S  chang°:  to  military  utili'-y.  Exo^rince  ini;ca‘'°5  ^ . 
such  techniques  lack  widespread  upo. 

C.  Issue  h?.  Exc‘=rie"ce  :f  ^"C£‘"am  '•',^3-0-, 
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oe  logical  10  oeiieve  tnat  tne  program  manager  (<16013100  maker)  must  nave  a 
tnorougn  unoerstanding  of  the  intended  system  use.  What,  tnen,  will  be  the 
effects  of  severely  restricted  entry  of  personnel  with  operational  experi¬ 
ence  into  tne  program  management  career  field  because  of  continual  opera¬ 
tional  personnel  snortages? 

liscussion .  A  reasonable  concl.usion  of  tnis  study  is  tnat  because  the 
military  program  manager,  like  the  non-miiitary  program  manager,  must 
strive  for  product  success,  knowledge  of  the  system's  ^product's;  intended 
use  13  vital  in  program  definition,  planning,  and  control.  To  be  success¬ 
ful,  a  program  manager  must  above  all  else  be  able  to  tnoroughly  articulate 
the  system's  end  use;  with  this  understanding,  programmatic  structure  anc 
decisions  can  be  properly  defined  and  defended  through  every  level  of  tne 
acquisition  organization. 

As  tne  entry  of  0-3/0-4  operationally  experienced  personnel  into  AFSC 
decreases,  how  will  program  managers  be  developed?  Program  management  is  a 
ccmpl.ex  process,  and  a  program  manager  must  have  a  reasonaole  amount  of 
experience  in  making  the  acquisition  process  work  to  be  effective  as  a  0- 
5/0-O  leve^  manager.  In  raising  this  particular  issue,  tr.e  study  team 
recognizes  that  the  proper  specialty  mix  in  AFSC  program  management  has 
oeen  a  continuing  concern  of  Air  Force  personnel  managers.  The  issue, 
however,  is  exacerbated  by  the  study's  findings  of  now  important  the  pro¬ 
gram  manager's  operational  knowledge  and  understanding  is  to  tne  success 
of  the  program. 

C.  Issue  (^3.  Requirements  Process. 

^e.3t .  in .  Given  the  relatively  low  correlation  between  successful 
programs  and  "the  existence  of  clear  requirements"  (see  section  IV-D  for 
letaiij/,  are  mere  correctable  deficiencies  witnin  the  Air  Force  req^ire- 
ments  process  wnicn  impede  system  program  management  success? 
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Discussion 


The  requineaents  process,  central  to  a  procrar'?  P»^Jin- 
ning  ["What's  needed?",'  and  eventual  deplovnaent  '"Does  the  systec  ree'  th>^ 
requirement?"',  has  Peer  reviewed  on  numerous  occasions.  recause  tnis 
study  found  that  “•  of  the  ’ '  most  successful  programs  cccurred  in  the 
absence  of  clear  requirements,  and  hecau.se  at  least  three  programs  r-u , 
ur-'’''A,  and  ?DSS[  were  reported  by  respondents  to  hav>’  unclear  or  unreal¬ 
istic  requirements ,  the^'e  is  reason  to  believe  that  the  requirements  pro¬ 
cess  remains  flawed. 

One  clue  to  the  success  of  many  of  the  top-rated  programs  seems  to  be 
the  high  degree  of  user  Involvement  in  the  detailed  system  design.  This 
"Try  Before  Buy"  approach  seems  to  be  quite  useful  because  t.'"e  user  must 
invariably  and  inevitably  take  the  time  and  effort  to  balance  system  per¬ 
formance,  employment  concept,  and  operational  doctrine.  Without  having 
the  opportunity  po  gain  first-hand  use  in  the  system's  oapabi 1 i t .es ,  the 
user  does  not  nor  cannot  know  what  performance  the  system  really  needs  to 
do  the  Job.  Hence,  "requirements"  are  sometimes  unrealis*'ic  and.'cr  un¬ 
necessary  , 

The  value  of  full-scale  system  prototypes  to  demonstrate  technology 
achievement  and  to  enhance  cost  realism,  while  important,  may  overlook 
perhaps  the  biggest  benefit:  it  allows  the  user  to  determine  if  the 
i.ntended  '.or  perhaps  another  unintended'  military  missi-u  can  be  satisfac¬ 
torily  and  affordably  accomplished  by  the  system.  Any  new  lock  at  the  Air 
Force  requiremenos  process  should  acknowledge  the  effect  that  prototype 
evaluations  have  had  on  the  success  of  the  F-i6,  E-?A.  A-'C,  and  F-5t.  F 
programs . 


I 


f-i 


. . 


1 


CON’CL'JSICNS 


•  Stu'-V  Methodology 

’  •  Vs6  Icpins Ir*^  P'^ovid'?^  rsspcr.^^c  t-hsr: 

oxpect.e'ti . 

2.  Tp.volverner.t  of  ipdustrv  ip  acquisition  rpspanch.  whi 
is  nestricted  by  factors  ouch  as  corporate  policy,  fear  of 
current/totential  p'jstoner.  or  lack  of  financial  iromtive  tO' 

3 .  Success  in  System  Program  Management 

little  effort  is  tiade  in  Eanageaent  literature  to 
in  program  management.  Success  is  used  tc  mean  ‘'ulfilling 
of  the  onganizaticn. 

2.  In  the  private,  non-military  sector,  program  success  usually  is 
equated  with  produc*"  success,  "he  basic  method  of  evaluating  a  program  is 
the  sire  of  the  financial  return  from  the  resultirg  product.  Hence, 
program  managers  are  very  knowledgeable  of  and  sensitive  to  ‘'h'’  expected 
rrcduct  I’ae. 

3.  Within  the 
by  ;FSC?  800-3  as  a 
suggest  that  thes" 

■■'is/her  "mission." 

u.  Hypothesis  I.  "fvs‘'em  rrogrnm  ruco't'sr  is  defined  in  "erms 
the  program's  rec"""d  "  ""ptinv  di-""ted  cost/ sr'''’!:'ui  °/oer''o''''.-'n''e 
'’equi  "ements ,  rnd  'b'  r^ows.n  t  rv''‘‘‘’T's  ebiii*:','  ‘'c  atta  '■  s’",'' 

*13  ’  n  3  rr  ^  1  ^3  V  O  ^  fm  <t  c;  ”  W'*,  "’J  ^  ^  a  •-  P  e  ^  ~  p 

*’’^3*1  3  t''-' i7 ‘"s ^  ^  ■  'j d Pd  r ^  3  c* - p  ,  w, ^ , 

well  ^ollow*nc^  ?ridt^r’3  '  ''■"rV'-  '-•r  *■  • 


Air  "orre  ^rvironaiend ,  orocraT  '^ar.aj^enipr.t  is  described 
proo‘='SS  or  ocm.clet:  ng  orog^am  objectives,  feme  authors 
ob'oo^iv'^''  0"'='  p.asseo  dov,"'^  to  the  r/rogram  manager  as 


define  success 
dhe  object-iver 


le  desirable, 
alienating  a 
pn-'dio:  pate . 
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a.  The  system  was  operationally  capable. 

b.  The  program  met  its  cost/schedule/performance  objectives. 

0.  The  program  maintained  stability  of  program  requirements. 

d.  Deployment  of  the  system  was  timely. 

e.  There  were  sufficient  numbers  of  the  system  to  be  militarily 

significant. 

f.  The  program  and  personnel  received  awards  or  promotions. 

5.  Hypothesis  II,  "The  closer  one  is  to  the  Program  Office  (in  an 
organizational  sense) ,  the  more  likely  it  is  that  success  will  be  defined 
in  terms  of  meeting  CSP  requirements,  "  was  not  substantiated.  The  two 
most  important  factors,  irrespective  of  organizational  perspective,  appear 
to  be  (a)  Deploying  a  capable  system  and  (b)  Producing  a  system  for  which 
the  operational  requirement  has  been  clearly  established. 

6.  The  five  major  Air  Force  system  programs  felt  by  respondents  to  be 
the  most  successful  during  the  period  1965-1980  were: 

a.  F-15 

b.  C-141 

c.  F-16 

d.  Mlnuteman  II  and  III 

e.  F-5E/F 

7.  All  causes  of  success  were  classified  under  these  five  main  input 
parameters: 

a.  Attention  to  directed  CSP  requirements. 

b.  Ability  to  maintain  tight  control. 

c.  Adequacy  of  means  to  accomplish  program  objectives. 

d.  Support  for  the  system  program. 

e.  Existence  of  clear  requirements. 


8.  Using  the  success  criteria  developed  in  Phase  I,  the  most  success¬ 
ful  Air  Force  programs  of  those  evaluated  in  the  period  were 

determined  to  be: 

Most  Successful  Moderately  Successful 
C-141 

AGM-b5  Maverick 
: -bE/E 
F-lb 
E-3A 

9.  Hypothesis  III,  "System  program  success  is  impeded  because  the 
Program  Manager  is  disciplined  to  meet  CSP  requirements  and  to  resist 
changes  which  may  be  necessary  to  ensure  long  term  military  effectiveness 
of  the  system,"  was  not  substantiated.  Collected  data  indicated  that  the 
desire  for  system  effectiveness  exceeded  that  for  adherence  to  CSP  re¬ 
quirements  . 

'0.  Hypothesis  IV,  "The  most  im.portant  determinant  of  system  program 
success  is  the  clarity  and  understanding  of  the  intended  mission,"  could 
not  be  substantiated.  Cne  input  sub-parameter,  "Availability  of 
"Technology,"  appeared  to  have  the  most  importance. 

IT.  The  highest  priority  causes  of  system  program  success  are  avail¬ 
ability  of  technology  and  strict  adherence  to  system  performance .  A  review 
of  the  most  successful  programs  revea-ed  various  methods  to  enhance  these 
causes : 

Prototype  to  demonstrate  technology. 

Extensive  rceratcr  input  ur.  testing. 

Use  of  of  f-th^-sh.eif  compcnents . 

Follcw-on  from  a  maiure  system. 

tune  ays'em. 


a. 


Cunship 
F- '  5 
SRAM 
A-7 
A- 10 

:<c-io 


e.  '^edification  of  a  ma 


/ 


12.  System  program  success  tends  to  be  measurea  in  terms  of  how  affir¬ 
mative  and  strong  the  answers  are  to  the  following  questions: 


a.  Does  the  user  understand  what  to  do  with  the  system  to  obtain 
military  utility? 

b.  Does  the  system  work  well? 


2.  Analyze  the  effects  and  possible  corrective  actions  concerning  the 
increasingly  restricted  entry  of  operationally  experiencec  personnel  into 
tne  acquisition  aanagement  career  field  because  of  personnel  shortages. 

3.  Review  tne  Air  Force  requirements  process  to  recommend  better 
methods  for  developing/deploying  systems  with  significant  military 
utility . 
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Survey  Questionnaire 
To  Support 

Air  Force  Business  Research  Management  Center  (AFBRMC) 
Study  of  Successful  System  Program  Management 


Conducted  by 


Advanced  Technology,  Inc, 
Contract  F33615-80-C  5184 


Instructions: 

1.  This  IS  a  five-part  questionnaire.  To  prevent  biasing  your  response  to  Parts  i-lll,  please  complete 
each  part  in  turn,  and  do  not  return  to  any  part  you  have  completed. 

2.  Kindly  use  the  enclosed  envelope  to  send  us  your  completed  questionnaire. 

3.  Do  you  desire  a  copy  of  the  study  report  which  will  be  based,  in  part,  on  your  response? 

Yes _  Mn 

4.  If  you  have  questions  concerning  the  questionnaire  or  study,  or  if  you  wish  to  provide  additional 
comments,  please  call  Mr.  Frederick  B.  Wynn,  703-521-9220. 


PRIVACY  ACT  STATEMENT 

In  accordance  with  Paragraph  8,  AFR  1 2-35,  Air  Force  Privacy  Act  Program,  the  following  information 
about  this  survey  is  provided: 

a.  Authority:  Federal  Statute  Title  10,  DSC  Section  80-12  Secretary  of  the  Air  Force:  Powers 
and  Duties,  Delegations  by. 

b  Principal  purpose:  This  survey  is  being  conducted  to  seek  a  meaning  of  success  across  all 
programs  and  ways  in  which  future  programs  can  be  made  successful. 

c  Routine  use:  Survey  data  will  be  compiled  and  analyzed  as  a  data  base  for  research  into 
determining  criteria  for  successful  acquisition  programs. 

d.  Participation  in  this  survey  is  entirely  voluntary. 

e.  .No  adverse  action  of  any  kind  may  be  taken  against  any  individual  who  elects  not  to 
participate  in  any  or  all  of  this  survey. 

USAF  Survey  Control  Numoer  81  -1 1 


Part  I.  To  whatever  depth  you  'e. 
successful  system  program  ma’’  . 


Part  II.  From  the  list  of  major  programs  attached  to  this  questionnaire,  select  three  you 
believe  were/are  most  successful,  and  comment  why. 


d.  Program _ 

Successful  because: 


b.  Program _ 

Successful  because; 


c.  Program _ 

Successful  because; 


Part  III.  From  the  list  of  major  programs  attached  to  this  questionnaire,  select  three  you 
believe  to  be  least  successful,  and  comment  why. 


a.  Program _ 

Least  successful  because; 


b.  Program _ 

Least  successful  because; 


c.  Program . . 

Least  successful  because; 


Part  IV.  Column  A  lists  factors  which  may  partially  define  system  program  success.  Please 
select  those  factors  from  most  to  least  important  and  rank  them  in  Column  B  (1  is  most 
important,  2  is  second  most  important . 10  is  least  important). 

COLUMN  A.  "A  SUCCESSFUL  SYSTEM  COLUMN  B.  IMPORTANCE 

PROGRAM  IS  ONE  WHICH:  . .  OF  EACH  FACTOR 

a.  Minimizes  the  time  from  need  identification  to  introduction 
into  operational  use. 

b.  Integrates  support  and  manpower  concerns  into  the  acquisi-  .  — .  .  .  - 

tion  process  such  that  system  deployment  is  accomplished 

with  minimum  disruption  to  the  operational  force. 

c.  Clearly  defines  an  acquisition  strategy  at  program's  outset  _ 

and  closely  adheres  to  it  throughout  the  life  of  the  program. 

d.  Meets  directed  (Decision  Coordinating  Paper/Program  - 

Management  Directive)  requirements  for  Cost/Schedule/ 

Performance/Supportability. 

e.  Produces  a  system  for  which  the  operational  requirement 
has  been  clearly  established. 

f.  Defines  and  attains  affordability  goals  for  each  acquisition  _ 

phase. 

g.  Deploys  a  system  with  capability  which  is  competitively  _ 

effective  against  the  threat  during  the  system's  projected 

life. 


h.  Incorporates  the  most  current  state-of-the-art  technology  in 
the  system  design. 

i.  Minimizes  design  changes  throughout  development  and 
production/deployment. 

j.  Meets  an  established  Initial  Operational  Capability  (IOC) 
date. 


MAJOR  AIR  FORCE  SYSTEM  PROGRAMS 


System  programs  are  defined  as  "major"  and  included  on  tiiis  list  by  meeting  any  of  the  following 
criteria  for  the  1965-1980  period: 

1 .  Selected  Acquisition  Report  (SAR)  program 

2.  Designated  Systems  Management  Group  (DSMG)  program 

3  Program  Assessment  Review/Secretary  of  the  Air  Force  Program  Review  (PAR/SPR) 
program. 
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A-10 

Advanced  Ballistic  Re-Entry  System  (ABRES) 

Air  Force  Satellite  Communications  System  (AFSATCOM) 
ATC  Radar  Beacon  IFF  Mark  XII  System  (AIMS) 

Air  Launched  Cruise  Missile  (ALCM) 

Advanced  Medium  STOL  Transport  (AMST) 

Aero-Propulsion  Systems  Test  Facility  (ASTF) 

8-1  (Advanced  Manned  Strategic  Aircraft,  AMSA) 

C-5A 

C-141 

C-141  Stretch 
OOD  COMSAT 

OOD  Space  Transportation  System 
DroneiRemotely  Piloted  Vehicle 

Defense  Satellite  Communications  System  (DSCS)  I  and  II 
Defense  Support  Program  (DSP) 

E-3A  (Airborne  Warning  and  Control  System,  AWACS) 

E-4  (Advanced  Airborne  National  Command  Post,  AABNCP) 

ef-iha 

F'RF4 

F-5  E/F  (International  Fighter  Aircraft,  IFA) 

F-15 

F-;6 

FRF/FB-111 

Ground  Launched  Cruise  Missile  (GLCM) 

Gunship 

KC-IOA  (Advanced  Tanker/Cargo  Aircraft,  ATCA) 

Maverick 

Minuteman  II  and  III 

NAVSTAR  Global  Positioning  System 

Precision  Action  Link  (PAL) 

Pave  Strike 

Precision  Location  Strike  System  (PLSS) 

Program  647  (CLASSIFIED) 

Sabres  (Self  Aligning  Boost  Re-Entry) 

Space  Defense  System  (SDS) 

Simulators 

Short  Range  Attack  Missile  iSRAM) 

Titan  III 


Attachment  to  Survey  Questionnaire 
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SURVEY  RESULTS 


TABLE  I.  CLASSIFICATION  OF  SUCCESS  CRITERIA 

Successful  System  Program 

1 .  Accomplished  Program  Objectives 

1.1  Met  Cost/Schedule/Performanoe  (CSP)  Requirements 

1.1.1  Fulfilled  DCP/PMD  CSP  Requirements 

1.1.2  (Industry  Viewpoint)  Met  contract  requirements 

1.1.3  Completed  well  planned  Test/Eval  program 

1.1.4  Provided  good  "audit  trail" 

1.2  Maintained  Program  Stability 

1.2.1  Well  defined  program 

1.2.2  Performance  requirements  constrained 

1.2.3  Changes  minimized 

1.2.4  Risks  minimized 

1.2.5  Risk  continually  assessed  and  controlled 

1.2.6  Program  schedule  "sacred,"  unvarying 

1 . 3  Cancelled  program  when  obvious  that  system  would  not 
satisfy  military  mission. 

2.  Program  and  Personnel  Rewarded 

2.1  Program  and/or  personnel  received  awards 

2.2  Personnel  promoted 

3.  System  Worked  Well  When  Needed 

3.1  System  was  capable 

3.1.1  Met  user  needs 

3. 1.1.1  High  operational  utility 

3. 1.1. 2  High  operational  supportability 

3.1.2  Good  design 

3. 1.2.1  System  worked  well 

3. 1.2. 2  System  adaptable  to  new  uses 

3. 1.2. 3  Real  trade-offs  were  made 

3.1.3  Contractor  available  to  support  system 

3.2  System  deployment  was  timely 

3.2.1  Development  time  was  shortest  possible 

3.2.2  System  equipment  provided  before  need  date 

3.3  System  numbers  were  sufficient 


TABLE  II.  IMPORTANCE  OF  SUCCESS  CRITERIA 


Criteria 

.  (to 

third  level 1  Total  Responses 

Importance 

Successful  System  Program 

92 

N/A 

1. 

Accomplished  Program  Objectives 

42 

High 

1.1 

Met  CSP  Requirements 

25 

Moderate 

1.2 

Maintained  Program  Stability 

16 

Moderate 

1.3 

Cancelled  program  when  obvious 

system  would  not  satisfy 

military  mission 

1 

Some 

2. 

Program  Personnel  Rewarded 

0 

None 

2.1 

Received  Awards 

0 

None 

2.2 

Personnel  Promoted 

0 

None 

3. 

System  Worked  Well  When  Needed 

50 

Highest 
(of  2nd  Level) 

3.1 

System  Was  Capable 

42 

Highest 
(of  3rd  Level) 

3.2 

Deployment  Was  Timely 

7 

Some 

3.3 

System  Numbers  Sufficient 

1 

Some 

Importance  Standards: 
Standards  Responses 
None  0 

Some  1-15 

Moderate  16-30 

High  30+ 
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TABLE  III. 


EXAMPLES  OF  SUCCESS  DESCRIPTORS 


Successful  System  Program 

1.  Accomplished  Program  Objectives 

1 . 1  Met  CSP  Requirements 

1.1.1  Fulfilled  DCP/PMD  CSP  Requirements 

1 .  Executing  directed  program  to  meet  CSP 
Requirements 

2.  PM  satisfies  CSP 

3.  Management  delivered  what  it  promised 

4.  Completed  program  by  IOC  within  CSP 

5.  Delivering  system  within  specified  Schedule 
and  Performance 

6.  Lives  up  to  DCP  promises  made  to  OSD 

7.  Met  both  operational  and  cost  objectives 

8.  Minimal  cost  to  bring  system  into  inventory 

9.  From  SPD’s  perspective:  Meet  CSP  Requirements 

10.  Survey  factor  IV.d. 

11.  Survey  factor  IV.f. 

12.  Ultimate  in  successful  management  is 
producing  to  schedule  and  performance 
within  cost. 

13-  Cost/schedule  met. 

14.  Costs  were  not  excessive. 

1.1.2  (Industry  viewpoint)  Met  contract  requirements 

1 .  Met  contractual  CSP  requirements 

2.  Equitable  contract 

3.  Satisfaction  of  buyer  and  seller 

4.  Operational  system  that  meets  customer's 
requirements 

5.  Contract  meets  profit  objectives 

6.  Excels  over  competition 

7.  Minimizes  unrewarded  investment 

1.1.3  Completed  well  planned  Test/Eval  program 

1 .  Followed  well-designed  TiE  program 

2.  Test  programs  validate  hardware 
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1.1.4  Provided  good  "audit  trail" 

Program  allows  a  good  "audit  trail" 

1.2  Maintained  Program  Stability 

1.2.1  Well  defined  program 

1 .  Had  one  basic  requirement 

2.  The  program  was  initially  well  defined 

1.2.2  Performance  requirements  constrained 

1 .  Stable  requirements  base 

2.  Performance  requirements  were  kept  constant 

1.2.3  Changes  minimized 

1 .  Air  Force  was  sold  on  original  requirements 

2.  Miminum  changes  through  production 

3.  Requirements  remained  firm 

4.  The  requirements  didn't  change 

5.  Survey  factor  IV. c. 

6.  Survey  factor  IV. i. 

1.2.4  Risks  minimized 

1.  Only  tried  what  could  be  done 

2.  Scate-of-the-art  program 

3-  Program  only  completed  if  CSP  goals  possible 
4.  Reducing  objectives/goals  until  useful 
product  realized 

1.2.5  Risk  continually  assessed  and  controlled 

Continuing  risk  assessment  and  control 

1.2.6  Program  schedule  "sacred,"  unvarying 

Relatively  firm,  unvarying  schedule 

1.3  Cancelled  program  trtien  obvious  that  system  would  not 

satisfy  military  mission 

2.  Program  and  personnel  rewarded 

2.1  Program  and/or  personnel  received  awards 

2.2  Personnel  promoted 

3.  System  Worked  Well  When  Needed 

3.1  System  was  Capable 

3.1.1  Met  user  Needs 

1.  Meets  the  user's  needs 
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2.  Meets/exceeds  the  user's  requirements 

3.  Reliable  and  maintainable  hardware 
Continued  essential  deterrence 

5.  Met  urgent  operational  needs  with  reasonable 
costs 

6.  Survey  factor  IV. g. 

7.  Achievement  of  usable  weapon  system  despite 
environment  in  which  program  is  undertaken 

8.  Optimum  system  performance 

9.  Quantum  Jump  in  air  superiority 

10.  Unifying  force 

11.  Significant  improvement  over  predecessor 

12.  Recognition  of  capability  by  users 

13.  Effective  weapon 

3 .1.1.1  High  operational  utility 

1.  Providing  system  which  satisfies  need 

2.  Demonstrated  operational  utility 

3.  Fulfilled  major  operational  need 

11.  Affordable  capability  to  satisfy  need 

5.  Satisfied  a  real  need 

6.  Insure  system  is  combat  capable 

7 .  Program  of  national  importance 
3.  Survey  factor  IV. h. 

9.  Survey  factor  IV. e. 

3. 1.1.2  High  operational  supportability 

1.  Demonstrated  operational  supportability 

2.  Survey  factor  IV.b. 

3.  Depot  capability  at  IOC 

3.1.2  Good  design 

1.  Well  defined  system  design 

2.  Excellent  engineering  system  design 

3.  Straightforward  engineering  design 
11.  Good  design 

Recognition  of  great  advancement  in  state- 
of-the-art 
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6.  Develop  from  state-of-the-art  technology 

3. 1.2.1  System  worked  well 

It  ^ohe  system)  works 

3. 1.2.2  System  adaptable  to  new  uses 

1 .  Design  adaptable  to  changing  needs 

2.  Retained  built-in  flexibility 

3.  Capable  of  growth 

4.  Provide  for  appropriate  growth 

5.  Designed  in  adaptability 

3. 1.2. 3  Real  trade-offs  were  made 

1.  Ability  to  balance  requirements  against 
resources 

2.  Enable  real  (as  opposed  to  predetermined) 
trade  studies 

3.  Achieving  best  compromises 

3.1.3  Contractor  available  to  support  system 

1 .  Contractor  was  covered  for  total  system 
support 

2.  Provide  continuing  support 

3.2  System  Deployment  Was  Timely 

3.2.1  Development  Time  was  shortest  possible 

1 .  Shortest  possible  time  for  development 
(don't  drag  out) 

2.  Survey  factor  IV. a. 

3.  Timely  deployment  of  much  needed  system 
capability 

4.  Quick  response  to  urgent  needs 

3.2.2  System  equipment  provided  before  need  date 

1 .  Equipment  provided  before  need  date 

2.  Put  system  into  use  in  reasonable  time 

3.  Survey  factor  IV. J 

3.3  System  Numbers  Were  Sufficient 

Produced  useful  systems  at  cost  to  allow  enough  to 
be  bought 
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Group  Identifier  d  cifhebjag 


A 

10 

9 

8 

7 

0 

5 

4 

3 

2 

1 

B 

9 

8 

7 

6 

6 

5 

5 

4 

3 

2 

C 

6 

6 

6 

6 

6 

5 

5 

5 

4 

4 

D 

4 

4 

5 

5 

5 

6 

6 

6 

6 

6 

E 

2 

3 

4 

5 

5 

6 

6 

7 

8 

9 

F 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

TABLE  V.  ASSUMED  RESPONSES  PER 
QUESTION  BY  POPULATION  GROUP 
(Assumed  if  Hypothesis  II  is  valid) 


Group  Identifier  dcifhebjag 


A 

5.8 

5.2 

6.8 

6.2 

8.0 

3.0 

4.2 

7.6 

5.4 

2.3 

B 

4.3 

4.3 

6.7 

6.7 

6.3 

3.0 

5.3 

8.0 

7.3 

1.7 

C 

3.0 

5.2 

7.0 

7.0 

8.0 

6.6 

5.2 

8.4 

3.0 

1.8 

D 

3.5 

3.5 

9 

6 

7.5 

1.5 

6 

9.5 

7 

1.5 

s 

4.0 

4.2 

6.1 

5.3 

8.6 

5.1 

5.3 

7.7 

5.1 

3.0 

F 

5.0 

5.0 

8.2 

6.4 

9.2 

4.6 

3.6 

5.8 

4.2 

1.8 

TABLE  VI.  AVERAGE  RESPONSE  PER 
QUESTION  BY  POPULATION  GROUP 
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Annex  I  to  Appendix  3  :  Statistical  Analysis  -fl 


Test  for  Hypothesis  II. 

Vsing  Groups  A,  C,  E 
Questions  d,  h,  g 

Assume  normal  distributioHjStudent  "t",  degrees  of  freedom  =  29, 
to  compare  two  means. 

Ho  ;  =  10  -tJAh  =  6  "^Ag  *  1 

.■qcd  =■  6  "  6  -<^Cg  =  9 

>HEd  =  2  =  5  -^Eg  =  9 

Vse  five  per  cent  level  test 


True 

values , 

95%  confidence 

t 

Ho  i 

■Accept  or 
Re j  ect 

3.99 

C  -UAd 

1  7.51 

'  1 

ilO  i 

Reject 

6.  25 

4  -UAh 

^  9.75 

!  ®  ' 

Rej  ect 

.23 

L  -UAg 

4  3.35 

^  : 

■  1 

i  : 

■Accept 

1.15 

4  -UCd 

^  4.35 

1 

:  ^  i 

t  ■ 

Rej  ect 

5.76 

^  10.00  (upper  limit  of  values) 

^  6 

Accept 

.45 

-  -Hf  § 

^  3.15 

4  ; 

Rej  ect 
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Annex  2  to  Appendix  3: 

Statistical  Analysis  ^Jo.  2  -  All  groups  respond  similarly. 
Using  an  Analysis  of  Variance  for  One-Way  Classification 


Ref;  Wine,  R.  Lowell  "Statistics  for  Scientists  and  Engineers," 
Prentice-Hall  Inc.,  1964, pp.  312-315 

Objective:  Select  question  IV  "d , "  analyze  to  determine  if  means 

of  the  groups  are  significantly  the  same. 

Data  Available; 


Groups  A 
Samples  5 


D  E 

2  15 


Analysis  Method;  To  increase  the  significance  of  the  test.  Combine 
Groups  into  Three  Blocks; 


Block; 
Groups ; 


I 

(A+3) 


II 

(C+D) 


III 

(E+D 


Hence,  Hull  Hypothesis  is 


Ho  *  Jti,.  =  Ai,.  j  =  -"tt  j 


Significance  level  o(,  ”  0.05 


Statist; 


r 

> 

>  ^05  ^2.32) 

*  3.30 

Sm”  * 

3  8,7,20 

£  ^  ( 

i=  1  i  =  1 

f.  .\2 

^xi.  -  xi; 

b-1 

s 

CO 

(xij  -  .xi.) 

6 

"l"^  "2  "3 

-  3 

n 

Sm”  * 

1 . 144 

Sm" 

9 

Se”  = 

6.57 

Se"" 

There 

is  reason  to 

believe  that 

if 


Sm 

Se' 


'.05’ 


rej  ect 


.174 


30 
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SELECTED  PARTS  I 
FROM  SURVEY  QUESTIONNAIRE 


Introduction.  Part  I  of  the  Survey  Questionnaire  was  purposely  structured 
to  be  open-ended,  allowing  the  respondent  to  address  "success"  from  a  wide 
range  of  views.  Data  from  all  questionnaires,  including  Parts  I,  formed 
the  basis  for  the  final  report  of  this  study.  Some  of  the  Part  I  respon¬ 
ses,  however,  showed  so  much  insight  that  they  are  repeated  in  this  appen¬ 
dix  almost  verbatim. 

Comment .  Most  Parts  I  described  success  in  terms  of  either  the  "classical 
definition":  Bringing  the  system  into  the  inventory,  on  schedule,  within 
cost  estimates,  and  meeting  specifications;  or  from  a  user's  perspec¬ 
tive:  That  management  process  which  provides  to  the  operational  forces  a 
system  which  satisfies  a  true  operational  need. 

Responses  which  expand  on  these  themes  follow,  listed  generally  on  the 
basis  of  some  significant  viewpoint: 

a.  Working  definition  from  a  SPD. 

b.  Acquisition  strategy  based  on  system  (not  necessarily  program) 
requirements . 

c.  Characteristics  of  a  successfully  managed  system  program. 

d.  The  contractor. 

e.  "Micro"  program  management  (what  the  SPD  can  control)  vs.  "Macro" 
(what  the  SPD  cannot  control). 

f.  OSD 

g.  Importance  of  the  environment  and  factors  beyond  the  control  of 
the  SPD. 

h.  Making  certain  the  system  works. 

1.  Plea  from  a  user. 
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A.  Working  definition  from  an  SPD. 

"Successful  program  management  will  have  existed  if  the  following  con¬ 
ditions  are  met. 

1 .  System  capabilities  provided  the  using  cotmnand  are  equal  to  or 
greater  than  those  specified  in  the  requirements  documents. 

2.  The  costs  are  within  5%  of  the  estimates  in  real  dollars  for  total 
program.  (1-2%  in  annual  estimates). 

3.  The  IOC  is  within  6  months  to  a  year  of  the  need  date. 

4.  The  necessary  support  equipment  2uid  facilities  are  available  at 
IOC  and  included  in  the  cost  estimates. 

It  is  possible  that  successful  management  will  lead  to  termination  of  a 
program  if  the  above  conditions  cannot  be  met  due  to  funding  constraints, 
threat  changes,  and  user  gold  plating  requirements." 

B.  Acquisition  strategy  based  on  system  (not  necessarily  program)  re¬ 
quirements. 

"Successful  system  program  management  is  the  ability  of  a  manager  to 
take  the  requirements  of  the  user  (customer)  and  then  define  a  pro¬ 
gram/strategy  to  design,  develop,  test  and  deploy  that  system  and  have  it 
meet  the  requirements. . .those  requirements  must  remain  constant  (but)  if 
there  are  changes,  the  cost  and  schedule  impacts  must  be  recognized  and 
funded  without  prejudice  to  the  program." 

C.  Characteristics  of  a  successfully  managed  system  program. 

"A  successfully  managed  system  program  consists  of  a  number  of  diverse 
characteristics.  These  would  Include; 
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a.  Early  establishment  of  an  'accepted'  Requirements  Base  Line.  This 
often,  in  my  experience,  did  not  exist  until  a  Program  Manager  was  as¬ 
signed.  There  is  a  need  to  coalesce  the  often  loose  objectives  into  a 
meaningful  requirement  that  ultimately  can  become  an  unambiguous  contrac¬ 
ted  work  statement  and  well  defined  specification, 

b.  The  objectives  must  be  articulated  to  the  user,  AFLC,  and  the  Air 

Staff,  to  achieve  energetic  and  total  Air  Force  support  -  early. 

0.  Establishment  of  an  early  and  realistic  cost  Base  Line. 

d.  Clearly  articulated  to  DOD  and  Congress  to  obtain  their  support. 

e.  A  sound  business  contract  which  subsequently  cannot  be  second- 
guessed  by  antagonistic  critics. 

f.  Once  the  program  development  and  production  milestones  are  estab¬ 
lished,  it  is  essential  that  each  be  met. 

g.  Whenever  'outside'  interference  (OSD  or  Congress  reprogramming) 
causes  adjustments,  these  should  be  avoided,  and  when  unavoidable,  they 
should  be  recorded  with  clearly  articulated  impact  statements.  The 
'interferer'  should  be  identified  and  held  responsible, 

h.  Operational  performance  objectives  should  be  demonstrated  during 
test  evaluation, 

i.  Development  and  production  costs  should  coincide  with  original 
estimates.  This  is  more  difficult  with  double-digit  inflation,  but  infla¬ 
tion  impacts  should  be  isolated. 

J.  Original  I.O.C,  dates  should  be  achieved. 

Achieve  these  -  and  the  program  will  be  successful!" 

D.  The  contractor. 
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Teat  programs  meet  objectives  and  validate  hardware. 

-  Development  coats  are  close  to  estimates. 

-  Development  schedule  is  achieved. 

-  Production  cost  is  consistent  with  earlier  estimates. 

-  Program  must  be  perceived  by  OSD  and  Congress  as  being  'highly 
successful'  and  bring  credit  to  The  Service. 

-  The  system  meets  technical  requirements. 

-  The  system  is  satisfactory  to  the  user. 

RAM  characteristics  are  acceptable. 

-  Contractor  interfaces  are  responsive  to  Government  factors. 

-  Effective  support  is  sustained  at  least  within  the  military  and  at 

DOD. 

-  Meets  profit  objectives. 

Utilizes  resources  effectively. 

-  Minimizes  unrewarded  investment. 

-  Excels  over  Che  competition. 

Shows  understanding  of  2ind  meets  Government  'special'  objectives 
(i.e.,  any  not  overtly  stated  in  program  documents)." 

E.  "Micro"  program  management  (what  the  SPD  can  control)  vs.  "Macro" 
(what  the  SPD  cannot  control). 

"Let  me  begin  by  establishing  my  terms  of  reference  for  system  program 
management.  In  my  judgment  there  is  'micro'  program  management  (System 
Program  Office  level)  and  there  is  'macro'  program  management  (the  whole 
schmeer  for  lowliest  enlisted  or  civil  servant  participant  to  the  Congress 
and  even  the  Executive  Department  if  they  involve  themselves).  It  is  my 
opinion  that  to  achieve  (the  purpose  of)  this  research  you  only  want  to 
look  at  'macro'  program  management.  So  I  will  devote  myself  to  that.  And 
I  think  I  know  of  what  I  apeak.  I  was  a  System  Program  Director  and  I 
succumbed  to  the  'micro'  idea  that  the  SPD  is  responsible  for  everything. 
But  in  retrospect  I  realize  that  some  of  my  frequent  frustrations  resulted 
because  I  was  saddled  with  things  over  which  I  had  no  control.  So  the 
'micro'  view  is  terribly  fallacious  because  it  covers  only  a  fraction  of 
the  pivotal  decision-making  life  span  of  the  program. 
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In  contrast,  'macro'  system  program  management  covers  every  phase  of 
the  system  from  a  conceptual  cradle  to  the  post  operational  salvage  grave. 
Every  participating  agency  having  a  responsible  role  to  play  must  play  the 
role  well  and  in  a  timely  manner.  No  Program  Director  could  have  overcome 
the  dual  burden  of  controversial  concept  and  controversial  selection  of 
tne  ...  contractor  for  the  TFX;  these  errors  were  made  in  the  Senior  Air 
Staff  and  at  the  Sec  Def  level.  No  Program  Director  could  have  overcome 
the  burdens  of  unrealistic  requirements,  total  package  procurement  manage¬ 
ment  and  ...  his  'supporting  Air  Staff;  these  errors  were  imposed  above 
the  SPO  level.  No  Program  Director  could  foresee,  least  of  all  overcome, 
the  decision  of  a  ...  President  in  cancelling  a  B-1  Program  that  was 
apparently  developing  satisfactorily.  None  of  these  are  within  the  sphere 
of  SPD  control  but  each  dramatically  influ’-nced  the  programs  in  question. 

It  is  with  this  'micro-macro'  differentiation  in  mind  that  I  suggest 
successful  system  program  management  consists  of  all  of  the  following  as  a 
minimum: 

a.  Clearly  stated,  fully  Justified  and  realistically  attainable  oper¬ 
ational  specifications  that  accurately  relate  the  user's  needs  for  ful¬ 
filling  his  role  in  supporting  national  objectives. 

b.  Translation  of  the  user's  operational  specifications  into  design 
criteria  and  technical  specifications  fully  within  demonstrable  state  of 
the  art. 

c.  Development  of  a  realistic  schedule  for  acquisition,  development 
testing,  production,  training,  and  logistics  support. 

d.  Development  of  realistic  life  cycle  cost  data  to  accurately  re¬ 
flect  the  development,  production,  operational,  and  support  costs  of  the 
system. 

e.  Disciplined  participative  management  in  full  cooperation  between 
the  Air  Force  Program  Iff  ice,  the  AFLD  Support  'Manager,  the  Contractor  and 


f.  Effective  support  and  representation  of  the  program  by  AFSC,  AFLC, 
USAF,  emd  OSD  staffs  at  all  levels. 

g.  Support  of  the  Congress  to  include  timely  approvals  and  appropria¬ 
tions  of  requested  funding. 

Successful  integration  of  these  essential  elements  with  myriad  others, 
both  predictable  and  unpredictable,  will  produce  the  desired  synergistic 
effect  -  the  'bottom  line'  criteria  for  succesful  system  program  manage¬ 
ment.  The  'bottom  line'  is:  Meet  the  program  schedule  within  esti¬ 
mated/budgeted  coats  with  a  system  that  meets  or  exceeds  the  user's  techni¬ 
cal  requirements  and  is  fully  supportable  in  the  operations,  training  and 
logistics  support  senses." 

F.  OSD. 


"Successful  system  program  management  produces  a  product  generally 
within  the  bounds  of  the  various  parameters  established  for  it  at  the 
outsat.  In  other  words,  it  lives  up  to  the  promises  made  to  OSD  thru  the 
DCP  process,  and  to  the  Congress  through  the  SAR  process.  These  'promises' 
are  essentially  informal  'contracts'  by  which  the  credibility  of  the  per¬ 
former  can  be  judged. 

If  the  product  cannot  in  fact  meet  all  the  goals  and  objectives  (they 
are  never  requirements),  then  successful  system  program  management  must 
either  stop  the  program,  or  sensibly  reduce  the  goals  until  a  useful 
product  can  be  realized.  Such  relaxation  of  goals  must  be  balanced  between 
cost,  performance,  maintainability,  and  schedules  —  with  none  being  con¬ 
sidered  'sacred'. 

Successful  system  program  management,  in  a  more  administrative  sense, 
provides  a  useful  and  intelligible  record  of  its  progress  towards  its  goals 
—  providing  an  'audit  trail'  which  not  only  demonstrates  its  own  achieve¬ 
ment,  but  provides  information  which  may  improve  estimates  and  projections 
for  future  programs. 


Successful  system  program  management  should  persist  through  the  useful 
life  of  any  major  program,  making  possible  the  updating  and  improving  of 
performance  and  the  reduction  of  both  production  and  maintenance  (and 
training)  costs.  Good  management  can  considerably  extend  both  the  useful 
life  and  the  total  production  quantities  of  most  types  of  systems,  as  well 
as  possibly  making  them  attractive  for  foreign  transfer/sale. 

Program  management  would  also  be  more  successful  if  it  considers  the 
potential  need  to  ’surge'  or  'mobilize*  the  production  of  either  complete 
systems  or  the  spares  and  related  consumables  associated  with  higher  oper¬ 
ational  tempos  or  combat  situations.  I  would  acknowledge,  however,  that  if 
such  capabilities  are  not  stated  as  goals,  then  the  manager  may  be  hard 
pressed  to  make  sensible  judgments  on  his  own. 

By  my  definitions,  successful  system  program  management  may  or  may  not 
lead  to  a  very  useful  product  in  retrospect.  If  the  established  program 
objectives  turn  out  to  be  inappropriate,  that  can  hardly  be  blamed  on  the 
system  program  management  unless  they  were  party  to  the  formulation  of 
unrealistic  initial  objectives." 

G.  Importance  of  the  environment  and  factors  beyond  the  control  of  the 
SPD. 


"The  basic  principles  of  successful  program  aianagement  are  not  secret. 
You  can  find  no  end  of  repetitive  studies  covering  the  same  ground.  The 
problem  comes  in  the  application  of  those  principles  in  an  environment 
where  so  many  elements  essential  to  program  success  are  beyond  the  control 
of  the  program  manager.  The  AF  has  a  long  history  of  success  in  managing 
system  acquisitions;  then  in  the  last  15-20  years  the  advances  in  technol¬ 
ogy,  plus  inflation,  multiplied  acquisition  costs  several  times  so  that 
cost  rather  than  operational  requirement  has  become  the  dominant  factor. 
Then  too,  McNamara  ushered  in  a  new  era  under  which  the  OSD  staff  sought  to 
change  the  world  of  system  management  and  suddenly  we  had  too  many  mana¬ 
gers,  most  of  whom  didn't  understand  the  process,  and  program  stability 
became  a  thing  of  the  past.  (Kissinger  in  his  book  'White  House  Years' 

'His  eager  young  associates  hid  their  moral  ccnvic- 
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comments  on  McNamara. 


tions  behind  a  seemingly  objective  method  of  analysis  which  obscured  that 
their  questions  too  often  pre-determined  the  answers  and  that  these 
answers  led  to  a  long  term  stagnation  of  our  military  technology.'). 

The  greatest  contribution  to  program  management  would  come  with  accep¬ 
tance  as  fact  that  (1)  we  can  never  predict  with  precision  the  problems  to 
be  encountered  and  the  costs  to  be  incurred  in  developing  and  producing 
something  that  has  never  been  built  before,  and  (2)  that  if  we  delay 
production  until  all  technical  advancements  are  proven  we  will  produce 
obsolesence.  If  a  capability  is  vital  to  our  national  defense,  we  must 
afford  it.  Such  concepts  as  'Design  to  Cost'  not  only  are  utter  nonsense, 
they  are  dangerous. 

There  is  a  difference  between  program  success  and  successful  program 
management.  Management  is  but  one  element  in  program  success,  although  a 
very  important  one.  Inadequate  funding  by  Congress  can  ruin  a  program, 
even  if  the  management  is  excellent.  Many  other  factors  can  bias  results. 
The  C-5A  will  never  be  classified  as  a  successful  program  but,  in  my 
opinion,  the  AF  management  was  good.  (To  come  to  this  conclusion  I  exclude 
from  the  grading  of  management  the  decision  to  procure  under  the  Total 
Package  Procurement  Concept  which  was  forced  on  AF  program  managers.  TPPC 
precluded  prudent  management  actions  which  could  have  balanced  poor  con¬ 
tractor  performance  and  benefited  the  program.)  Inflation  has  had  a  severe 
impact  on  program  success  in  recent  years  but  this  should  not  be  a  measure 
of  management  success  since  projected  inflation  rates  are  provided  by  the 
Executive  Branch  and  reflect  that  built-in  optimism. 

In  a  broad  sense,  the  ultimate  in  program  success  is  to  meet  perfor¬ 
mance  specifications  on  schedule  suid  within  cost  projections.  In  the  real 
world,  the  success  of  program  management  is  measured  against  how  close  the 
managers  come  to  meeting  those  three  goals  in  their  current  order  of 
priority;  that  is,  which  is  most  important  at  that  time,  achieving  perfor¬ 
mance  specifications,  meeting  an  IOC  or  staying  close  to  budgetary  limits. 
The  Minuteman  I  program  was  considered  a  success  because  it  came  in  on  or 
ahead  of  schedule  and  met  operational  requirements — the  priority  objec¬ 
tives  at  that  time.  Placed  in  a  different  era  with  unpredicted  inflation 
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and  an  emphasis  on  cost,  the  verdict  might  have  been  quite  different.  When 
cost  is  dominant,  the  manager’s  flexibility  is  limited.  He  can't  buy 
performance  or  improved  lOCs." 

H.  Niaking  certain  that  the  system  works. 

"Successful  program  management  means  more  than  guiding  a  hardware  pro¬ 
gram  over  the  hurdles  to  get  a  favorable  production  decision.  It  means 
fielding  a  system  that  can  perform  its  combat  role  when  and  where  required. 
Successful  program  management  means  'designing-in'  the  operational  perfor¬ 
mance  (range,  speed,  accuracy,  etc.)  and  logistics  supportability  from  the 
very  first  day  the  program  is  conceived.  It  means  establishing  the  tradi¬ 
tional  cost,  performance  and  schedule  criteria  for  success  plus  a  rela¬ 
tively  new  and  equally  important  criterion  -  logistics  supportability. 
Progress  toward  goals  for  reliability,  maintainability,  readiness,  man¬ 
power,  and  operating  and  support  costs  provide  the  benchmarks  for  measur¬ 
ing  the  success  of  this  criterion..." 

I.  Pleas  from  a  user. 

"Without  trying  to  write  a  treatise  on  the  subject  of  acquisition  man¬ 
agement,  a  task  for  which  I  am  not  at  all  qualified.  I'll  offer  a  thought 
or  two  from  the  perspective  of  an  operator  and  maintainer. 

Firstly,  I  believe  we  tend  to  over-soecify  our  requirements  in  terms  of 
design  and  performance  characteristics.  In  so  doing,  we  sometimes  push  the 
contractor  away  from  taking  his  best  shot  by  preempting  it  with  something 
less.  In  my  view  we  could  be  better  served  by  describing  our  requirements 
more  in  terms  of  the  mission(s)  to  be  satisfied  and  encouraging  the  con¬ 
tractor  to  innovate  to  include  the  use  of  commercially  proven  designs  where 
practical. 

Secondly,  I  feel  very  strongly  about  reliability  and  maintainability. 
Nothing  is  so  frustrating  as  tryi.ng  to  operate  in  combat  with  a  weapon 
system  that  requires  too  much  7LC  (Tender  'oOV"  and  Care)  by  high’/  quali¬ 
fied  technicians— we  never  have  enough  of  them.  I  don't  bilieve  we  end  up 
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with  these  kinds  of  systems  solely  because  we  insist  on  high  levels  of 
performance.  Too  often,  in  my  view,  it's  because  we  do  not  economically 
motivate  the  contractor  to  design  in  good  maintainability  and  reliability. 
This  is  a  well-recognized  principle  represented  by  the  mission  of  AFLC's 
Acquisition  Logistics  Division  but  too  often  given  lip  service  and  honored 
only  in  the  breach.  We  have  to  make  it  worth  the  contractor's  effort  up 
front  ...  to  design  in  reliability,  use  high  quality  components  and  keep 
the  field  level  maintenance  man's  task  firmly  in  mind." 
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STUDY:  ANALYSIS  OF  SUCCESS  IN  SYSTEMS  PROCRAM  MANAGEMENT 


Investigative  Worksheet  for  _  Program 

I.  General  Data: 

Date  of  investigation  _ 

Location  of  investigation  _ _ 

Investigator  _ 

Data  Source  _ 


Attributes  of  Success.  From  the  data  source,  indicate  numerically 
in  this  section  the  degree  to  which  success  criteria  were  met. 

A.  Accomplishment  of  Cost/Schedule/Performance  (CSP)  Requirements, 

1.  Success  in  meeting  cost  requirements: 

0 _  10 

Low  High 

Evidence/Comments ; 


2.  Success  in  meeting  schedule  requirements: 

0 _  10 

Low  High 

Evidence/Comments : 
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3.  Success  in  meeting  performance  requiremercs : 


0 _ 

Low 

Evidence/Comments ; 

Maintaining  of  Program  Stability 

1 .  Clarity  in  defining  program  objectives 

0 _ 

Low 

Evidence/Comments : 

2.  Success  in  minimizing  changes: 

0 _ 

Low 

Evidence/Comments : 

3.  Success  in  recognizing/managing  risk: 

0 _ 

Low 

Evidence/Comments : 

Rewards  to  Program  Personnel 

1.  Indications  that  program/personnel  won  awards: 

0 _ 10 

Low  High 

Evidence/Comments : 


20 

High 


20 

High 


20 

High 


20 

High 
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2.  Indications  that  program  personnel  were  promoted  as  a 
result  of  program  accomplishments: 


0 

10 

Low 

High 

Evidence/Comments : 

How 

Well  System  Worked  When  Needed 

1 . 

Indications  that  system  met 

user's  operational  needs: 

0 

10 

Low 

High 

Evidence/Comments : 

2, 

Indications  that  system  was 

supportable: 

0 

10 

Low 

High 

Evidence/Comments : 

3. 

Indications  that  deployment 

was  timely  (e.g.,  met  IOC) 

0 

10 

Low 

High 

Evidence/Comments : 
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III. 


4.  Indications  that  sufficient  numbers  of  system  were  deployed; 

0  _  10 

Low  High 

Evidence/Comments : 


Cause  of  Success.  From  the  data  sources,  indicate  numerically 
the  degree  to  which  input  parameters  were  met. 

A.  Input  parameter  1.0:  Attention  to  Directed  CSP  Requirements 

1.  Adherence  to  system  performance  requirements: 

0 _ 10 

Low  High 

Evidence/Comments : 

2.  Adherence  to  system  supportability  requirements: 

0 _ 10 

Low  High 

Evidence/Comments : 


3.  Adherence  to  program  schedule: 

0 _ 10 

Low  High 

Evidence/Comments : 
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4.  Adherence  to  program  cost: 

0 

_ ^0 

Low 

High 

Evidence/Comments : 

5.  Indications  of  meaningful  CSP  trade-offs: 

0 

10 

Low 

High 

Evidence/Comments : 

Input  parameter  2.0:  Ability  and  success 

tight  control. 

in  maintaining 

1.  Continuity  and  authority  of  PM: 

0 

10 

Low 

High 

Evidence/Comments : 

2.  Emphasis  on  change  control  management; 

0 

10 

Low 

High 

Evidence/Conments : 
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3.  Evidence  that  CSP  control  techniques  were  used  (C/SCSC, 
TPM,  CPM,  etc.): 

0 _ 10 

Low  High 

Evidence/Comments : 

C.  Input  parameters  3*0:  Adequacy  of  means  to  accomplish  program 
objectives. 


Availability  of  resources: 

a.  Personnel: 

0 

10 

Low 

High 

b .  Technology : 

0 

10 

Low 

High 

c.  Finances  (Funding): 

0 

10 

Low 

High 

d.  Time: 

0 

10 

Low 

High 

Evidence/Comments: 
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2.  Evidence  of  SPO  Capability/Excellence: 


0 

10 

Low 

High 

Evidence/Comments ; 

Input  parameter  4.0:  Support  for 

the  system  program 

Support  above  program  office: 

a.  HQ  USAF: 

0 

10 

Low 

High 

b .  OSD : 

0 

10 

Low 

High 

c.  Congress: 

0 

10 

Low 

High 

Evidence/Comments : 


Support  at  working  leval  of  SPO  (with  logistics  manager 
user,  contractor,  and  headquarters) : 


Abbreviations  and  Acronyms 


AFLC  Air  Force  Logistics  Command 

AFSC  Air  Force  Systems  Command 

ASD  Aeronautical  Systems  Division 

ASD( — )  Assistant  Secretary  of  Defense  for  — 

AU  Air  University 

CMD  Contract  Management  Division 

DCP  Decision  Coordiating  Paper 

DDC  Defense  Documentation  Center 

DLSIE  Defense  Logistic  Support  Information  Exchange 

DTIC  Defense  Tecf.nicai  Information  Center 

ESD  Electronic  Systems  Division 

JCS  Joint  Chiefs  >if  Staff 

MENS  Mission  Element  Need  Statement 

OSD  Office  of  Secretary  of  Derense 

PEM  Program  Element  Monitor 

PM  Program  Manager 

PMD  P-'Ogram  Management  Directive 

HOC  required  Operational  Capabilitv 

SAFA.  Secretary  tf  tne  Air  Force,  Acquisition  and  Logistics 
SAMSt  Space  and  Missile  Systems  Organization 

v.now  Space  Division  and  oal^istic  Missive  ..ffice; 
SMS  Senior  Master  Sergeant 

Statement's'  i:  '>perat.  ional  Need 
SOW  Statement  of  Work 

SPD  System  Prog.-'am  Director 

oPl  System  Program  Pfflce 

STCL  Short  TaKeoff  and  „andir.g 

TFX  .'act-cal  F.gr.te'-.  SnA.ncw.'.  .te.  jmt-  .--11.. 
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